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(54) An image processing apparatus and method, and an image evaluation device and method 



(57) In image processing according to the prior art; 
the important part of photographic image data (referred 
to herein as the object) could not be determined and- 
therefore required human participation. 

A computer 21 which is the core of image process- 
ing calculates an edginess which is an image variation - 
from a differential value of data tor adjacent picture el- 
ements in a step SA110, and determines object picture 
elements by selecting only images with a large variation , 
in steps SA120. SA130. As optimum parameters for 
contrast correction and lightness compensation are cal- 
culated from image data for object picture elements iri 
steps SA310-SA330, image processing indicators 



based on object picture elements are determined, and 
optimum image processing can be performed automat- 
ically. After summing a luminance distribution for each 
area of the image, which is a feature amount, while uni- 
formly selecting picture elements in a step SB110, a 
reevaluation is performed by a weighting determined for 
each area in a step SB120, and a luminance distribution 
strongly influenced by the luminance distribution of the 
photographed object is thus obtained with uniform sam- 
pling. After determining the intensity of this luminance 
distribution in steps SB130-SB150. the image data is 
converted in a step SB160, and image processing can 
therefore be performed with optimum intensity while re- 
ducing the processing amount. 
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Technical Field to which the Invention Belong : 

This invention relates to an image processing method wherein an optimum image processing is performed auto- 
matically on photograph image data such as digital photograph image, and the image is evaluated, to an image process- 
ing apparatus, an image evaluation method, an image evaluation apparatus. 

Related Background Art * 

~ vanous kinds of image processing may be performed on digital image data, i.e., in which processing: contrast may 
be increased: color may be corrected; or lightness may be corrected. This image processing can usually be performed 
with a microcomputer. An operator confirms the image on a monitor, the necessary image processing is selected, and 
image processing parameters are determined. 

In recent years various types of image processing techniques have been proposed, and are now having consid- 
erable impact. However, a human operator is still required when it is a question of which technique to apply, and to 
what extent it should be used. This is because it was otherwise impossible to determine which digital image data had 
to be subjected to image processing. For example, in the case of image processing to correct the lightness of an image, 
the screen is made lighter if it is dark on the whole, and is made darker if it is too light. 

Now, consider the case of a photographic image ol a person filmed at night, where the background is near to pitch- 
darkness but the person in the picture has been well photographed. If this photograph is automatically corrected, it is 
attempted to make the image brighter due to the fact that the background is pitch black, so the final image appears as 
if the photo was taken in the daytime. 

In this case, if a human operator is involved, he pays attention only to the person in the picture. If the image of the 
person is dark, it would bo mado a little brighter, conversely darkening would be selected if the effect of flash, etc , was 
too bright. 

Hence, there was a problem in the prior art in that a human operator had to participate to determine the important 
part (referred to hereafter as the "object") of a photographic image. 

However, even when the importance of the image is evaluated by some technique, the determination process is 
performed in picture element units, and varying the importance in real time causes an increase in computation. 

Disclosure of the Invention 

It is therefore an object of this invention, which was conceived in view of the aforesaid problems, to provide an 
image processing method which permits an important part of a photographic image such as a digital photograph image 
to be detected, and an optimum image processing to be automatically selected, to provide an image processing ap- 
paratus, and to provide a medium on which an image processing control program is recorded. 

In order to achieve the aforesaid object, this invention is an image processing apparatus into which photographic 
image data comprising dot matrix picture elements is input, and which performs predetermined image processing on 
the input data. The apparatus comprises an image processing acquiring unit which acquires the aforesaid photographic 
image data, an image processing indicator specifying unit which performs predetermined summation processing on 
picture elements and specifies an image processing indicator based on the acquired image data, and a processing 
unit which determines image processing contents based on the specified image processing indicator, wherein the 
aforesaid image processing indicator specifying unit comprises an object determining unit which determines picture 
elements having a large image variation amount to be those of the object, and the aforesaid processing unit determines 
image processing contents based on image data for picture elements determined to be those of the object and performs 
image processing on the determined contents. 

Herein, it is assumed that in the case of an image of a photographic image of a person, the person is usually 
photographed in the center of the field. Therefore, the person is brought into focus to give a sharp image. When the 
image is sharp, the outline part becomes clear, and the amount of image variation becomes large. As a result, there 
is an extremely high possibility that there will be no error if it is assumed that picture elements with a large image 
variation amount are those of the original object which has been brought into focus. 

In the invention thus comprised, in the image processing indicator specifying unit based on photographic image 
data comprising dot matrix picture elements acquired in the image data acquiring unit, predetermined summation 
processing is performed on picture elements. This summation processing may take various forms, and the image 
processing indicator is basically specified based on the summation result. In the processing unit, the image processing 
contents are determined based on the specified indicator, and the determined image processing is performed. 

This means that useful information about the corresponding image can be obtained by performing summation 
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!^? Sin9 ° n ^ phot °9 ra P hic ima 9e data, and image processing is performed on the data. In this way the 

S™?^" 9 -.'^?^ ^ SPedfied ^ 3CtUal phot °9 ra P hic im *9 e «> optimum image processing can be 
performed even without human intervention. 

•i™! P h ° to 9 ra P hic ima f data comprises dot matrix picture elements and image processing is performed in picture 
to b?t^?;,h I" 1 1". < * 18 * determini "9 P^^« •hn« having a large image variation are determined 
to be those of the object. Image processing contents are then determined in the processing unit based on image data 
fo we elements determined to be those of the object, and image processing is performed based on the determfned 

n^tin^^lT' ac f ordi "9 , ° this Mention, the determination of the object, which in the past required human partici- 

SSr!T^ rt i ' a ' !M ' ' li l Wh6r6by ° pt ' mUm ima 9 e Pressing can be perform io^uitaW " 

modrfying the image processing contents according to the object. y y 

not Jllvto f l 1 !^ 0 ' ^r" 9 an ima 9 e Passing indicator from actual photographic data may of course be applied 
miSS 1 Ut u ISO ,0 3 SYStem ° n thS meth0d - ,n SUCh a sense ' this inventi °n is also an image processing 

P^efsin^ ,ma9e ^P" 8109 ^ matriX piCtUre e ' ementS is input ' and P-dete?mhed image 

SSS^J . P erf0rm . d - ,h ' S ,ma9e P rocessin 9 meth °d comprising an image data acquiring step for acquiring the 
summon P 9raPh ' C ^ ^ a " ima9e P^ 655 '^ indic ^ spring step for performing a predetermined 
fndT c Ttn a PfOCeSS,ng ° n P ' Cture elements based on quired image data, and specifying an image processing 
nd.cator. and a processing step tor determining image processing contents based on the specified indicator, and per 
p IZ n^rT prOCeSS,n9 - wherein lne a,oresa ' d ima 9 e Pressing indicator specifying step comprises an object de- 
fining step where.n picture elements having a large image variation amount are determined to be those of the 
to Tru7» »7 T !? T* af ° r ! Said processin 9 s,e P' ima 9 e Pressing contents are determined based on image data 

Saoe o L~lT f T7t 10 ^ th ° Se ° f thG ° bjeCt> and ima9e P rocessi "9 » performed based on the determined 
.mage processing contents. In th.s case, the apparatus offers all the attendant benefits and advantages of the invention 

«t a „H 7 apparatUS tor do,ormi "i"9 an object and performing image processing method may bo implemented by a 

l oara , ™ 1 aPParatUS ! S T nti ° ned abOVG ' ° r may be '"corporated in another instrument which comprises such an 

2232 h !, 800,56 ° f thiS inVen,i ° n C ° VerS Vari ° US ,orms of implementation. It may also be imple- 

mented by hardware or software, and can be modified as necessary. 

When the apparatus for implementing the concept of this invention is implemented by software the invention 
i S th equ a,ly to media on ***** ,his software * recorded and which can be used in exact* the same way. In this 
Dhotoo ao S hr c Tm! 0 r If 80 3 recordin 9^edium whereon an image processing control program is recorded for inputting 
SSSnSL 9 data corn P" s,n 9 ** matnx Picture elements by a computer, and which performs predeteLned 
ZSCSTT 8 ^ 5 6 K nPUt *?: ThS ^ Pr ° 9ram C ° mpriSeS an ima 9* Processing indicator speeffying step for 
SSS?. I 683 Ph0,09raph,c ,ma 9 e da,a ' an "™9e processing indicator specifying step for performing prede- 
Z^dSIZ^E! 0 ™'" ° n PiC,Ure e ' emen,S and SPeCifyin9 3n image processin 9 ***** and a processing 
mpn,r ha l PT ° C f SS,nQ ,ndlCator s P ec «ying step comprises an object determining step which determines picture ele- 

9 3 , T ,ma ? e Varatl0n am ° Unt ,0 b8 thOSS ° f me ° biect - * nd in the aforesaid Processing step, image 
processing contents are determined based on image data for picture elements determined to be those of the object 

Si£S e JL ro £ 88n 2 18 performed OT the determined contents. In this case, the recording medium offers all the 
attendant benefits and advantages of the invention. 

an „ l\l re ( cordin 9 medium may of course be a magnetic recording medium, an optomagnetic recording medium or 
ml k? r r f crd,n9 medlum ma V fa e developed in the future. It will of course be understood that the medium 
~I ?h C ° P ^°, r S8COnd COPy ' and that 3 telec °mmunication line may also be used to supply the program. In this 
SSL^T 18 n ° d ' ff ! rence re 9 ardi "9 application of the invention. There is also no difference if the program is 
written on a semiconductor chip. »>»■■■ 

m ooL h H re ,? ^ difference as re 9 ards th e concept of the invention even if one part is software, and one part is imple- 
memed with hardware, or when it is in such a form that one part is stored on a recording medium which can be read 
wnen necessary. 

Photographic image data means image data obtained when it is attempted to take a photograph of a real object 
Image processing tries to correct images by comparing the images with the real objects from which they were acquired 

3n a „'r V H? ,0n apPMeS n0t ° ny ,0 natUral ° bjeCts but also to unmade ones. More specifically, this includes 

image data read by a scanner, or image data captured by a digital camera 

a f„^ ri ° U K ,eC . hn ,T GS may bS ""P*** 1 to determine the variation of picture elements in the object determining step. 
A further object of this invention is to give a specific example of this 

nf JL'r 6 P roce ( ssin 9 me,h od provided by this invention, in the aforesaid object determining step, the amount 
of variation of picture elements is determined based on a difference between adjacent picture elements 

Hence according to this invention, in the object determining unit where an image varialion amount is determined 
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the determination is performed based on a difference of image data between adjacent picture elements. When there 
is a fixed interval between picture elements as in the case of a dot matrix, the difference of data between adjacent 
picture elements is directly proportional to a first order differential. This difference can be taken as the variation amount 
of the image. I n this case the difference is regarded as the magnitude of a vector, and the vector may also be constructed 
by considering adjacent directions. 

According to this invention, only the difference of image data between adjacent picture elements is found. Com- 
putation is therefore easy, and the processing amount for object determination can be reduced. 

The determination of an object is of course not limited to this technique, and it is a further object of this invention 
to provide other examples. 

_ According to the image processing apparatus provided by this inve ntion, in the aforesaid object determining unit, 
the criterion lor determining whether or not there is a large image variation amount changes according to the position 
of the image. 

In the case of a photograph for example, a person is often photographed in the center, in this case it may be said 
that in order to determine image processing content, the picture elements to be determined as the object should be 
selected from the central part of the field. However it may be said that whether or not there is a large variation in the 
image depends on a difference from a comparison value, and there is no reason why such a value always has to be 
constant. 

Therefore, to determine whether or not there is a large image amount according to this invention, in the object 
determining unit, this criterion is altered depending on the position of the image, the criterion for each position being 
compared with the image variation amount of each picture element. 

Hence, according to this invention, the assessment of image variation changes depending on the position of the 
image, and a highly flexible determination which considers image composition is thus possible. 

The criterion can be altered in various ways. As one example, a certain trend may be ascertained, or alternatively, 
a trend which causes a change may be read from the image. 

A further object of this invention is to provide an example of the former. 

In the image processing apparatus provided by this invention, in the object determining unit, the aforesaid criterion 
is set lower for the central part than for the edges of the image. 

By setting the criterion lower for the center than for the edges, it is easier to determine the center part of the image 
as the object even if the variation amount at the center and at the edges is approximately the same. Therefore if there 
is an image of a person in the central part, the picture elements of this person will be determined as the object more 
frequently. 

According to this invention, a determination can be made which gives more weight to the center area of a photo- 
graph, and a large amount of image data can be effectively processed. 

A further object of this invention is to provide an example of the latter when the criterion is varied. 

Accorcjinc; to the image processing apparatus provided by this invention, in the aforesaid object determining unit, 
the above criterion is based on the distribution of the aforesaid image variation amount at different points on the image. 

Hence according to this invention, in the object determining unit, the distribution of image variation is found in each 
part of the image, and the aforesaid criterion is determined after finding this distribution. Subsequently a comparison 
is made with this criterion to determine whether or not the picture elements are those of the object. 

According to this invention, as the object is determined taking account of the distribution of image variation for 
picture elements, the image data can be treated flexibly. 

When the criterion is determined based on distribution, it may be considered that there is a high possibility of 
finding the object in a part where there are many picture elements with a large variation amount, and the criterion 
maybe set low. 

Alternatively, basic setting patterns may first be prepared according to a variation distribution pattern, and a basic 
setting pattern may then be chosen based on the detected distribution pattern. 

At the same time, assuming that the image processing indicator specifying unit comprises such an object deter- 
mining unit, the image processing contents may be determined based on image data which is determined to be that 
of the object, and image processing may then be performed on the determined contents, there being no limitation on 
the specific processing method employed. For example, a luminance distribution of picture elements determined to be 
those of the object is found, and if the luminance distribution range is enlarged in a predetermined proportion when 
the luminance distribution is narrow, image processing to increase contrast is performed. If the luminance distribution 
of the object seems dark on the whole, a correction may be made to make it lighter. The color distribution ol picture 
elements determined to be those of the object is found, and it is determined whether or not the grey balance is off. If 
it seems to be off, tone curves are used to modify the grey balance. 

Hence, the importance of the image has an effect even if the image data is summed in order to specify the image 
processing indicator However, even if the importance of the image is determined by some technique, the work is 
carried out in picture element units, so varying the importance of an image in real time implies an increase of compu- 
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tational amount. 

A further object of this invention is to consider the importance of photographic image data such as digital photograph 
images in relatively simple terms, and perform optimum image processing automatically. 

The .mage processing apparatus provided by this invention is an apparatus for inputting photographic image data 
comprising dot matrix p.cture elements, and performing predetermined image processing. This image processing ap- 
paratus compnses an image data acquiring unit for acquiring the aforesaid photographic image data, an imaqe process- 
mg .nd.cator specifying unit which performs a predetermined summation processing on picture elements based on this 
acquired image data and specifies an image processing indicator, and a processing unit which determines imaqe 
processing contents based on the specified indicator and performs image processing. The aforesaid image processing 
u^ZT^l^.^^ J 3 featUte am ° Unt Uniform Samplin 9 unit which determines an image pro cessing 

~S Sh ' * lea tUfe am ° Unt ° V8f 3 Wh ° ' e SCfeen " and 3 feature amount wei 9"ting reevalua 

un t wh ch reevaluates the feature amount sampled in the feature amount sampling unit with a predetermined weighting 
In the aforesaid image processing unit, the image processing intensity is determined based on the reevaluated feature 
amount, and image processing is performed with the determined intensity. 

According to the invention having the above construction, photographic image data comprises dot matrix picture 
elements, and in the feature amount uniform sampling unit, the feature amounts of picture elements are uniformly 
sampled over the whole screen. In the feature amount weighting revaluation un ,t. the feature amounts that are sampled 
in this feature amount uniform sampling unit are reevaluated with a predetermined weighting. Then, in the processing 
unit, the image processing intensity is determined based on the feature amounts that have been reevaluated in this 
way, and image processing is performed. 

In other words, as the sampling is uniform over the whole screen and a predetermined weighting is applied after 
sampl.ng. the feature amounts obtained as a result are different from what is obtained by uniform sampling over the 
wnoie screen. 

According to this invention the sampling in the sampling stage is uniform over the whole screen, so the computa- 
.onal amount ,s not too high. At the same time, by applying a prodotorminod weighting after sampling, irrelevant eval- 
uafon .s not made as it would be if the picture elements were merely sampled uniformly without weighting. The invention 
therefore provides an image processing apparatus in which optimum image processing can be performed automatically. 

It mil be understood that the technique of performing a uniform sampling in the sampling stage and applying a 
predetermined we.ght.ng thereafter, may be applied not onry to a real device but also to a system in both of whlh 
cases it has all the attendant benefits and advantages of this invention. As a specific example of the concept of this 
invention, when the image processing apparatus is implemented in terms of software, there naturally exist recording, 
media on which the software is recorded which can be used to perform the function of the invention 

The feature amount uniform sampling unrl uniformly samples feature amounts over the whole screen for deter- 
mining the image processing intensity. For this purpose, all picture elements over the whole screen can be sampled 
but it is notnecessary to sample all of the picture elements if the sampling is uniform. " 

A further object of this invention is to provide an example of the latter case. 

According to the image processing apparatus of this invention, in the aforesaid feature amount uniform sampling 

^Ll , af ° resa ; d ,eatUre am ° unts are sam P |ed ,or selected P*"* 5 elements after uniformly thinning out the picture 
elements according to predetermined criteria. h'^u'b 

According to this invention, by thinning out the picture elements according to predetermined criteria, the number 

ele^ente 6 8 tp bS P rocessed is 'educed, and the aforesaid feature amounts are sampled from the remaining 

Herein, the term 'uniform thinning" comprises the case where picture elements are selected at a fixed interval 
and the case where they are selected at random. 

According to this invention, as the picture elemenls are thinned out when the feature amounts are uniformly sam- 
pled, the processing amount is reduced. 

The sampled feature amounts are reevaluated by a predetermined weighting in the feature amount weighting 
revaluation .unit. The sampled feature amounts are in picture element units, but the weighting can be applied either 
to picture element units or to suitable aggregates of picture elements. 

A further object of this invention is to provide an example of the latter case. 

According to the image processing apparatus of this invention, in the aforesaid feature amount uniform sampling 
unit feature amounts are sampled in area units that are divided according to predetermined criteria, and in the aforesaid 
feature amount weighting revaluation unit, a weighting is sot for each area and the feature amounts are then reoval- 

The invention as formulated hereabove assumes weightings in area units of the image that are divided according 
to predetermined criteria. In the feature amount uniform sampling unit, feature amounts are sampled in these area 
units^ while .n the aforesaid feature amount weighting reevaluate unit, the feature amounts are reevaluated with 
weightings set for each area. 
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The division of these areas may always be constant, or it may be made to vary for each image. In this latter case 
the division method may be changed according to the contents of the image. 

According to this invention, as the weighting is made to vary for each area, the computation is relatively simple. 
Any type of weighting technique can be employed provided that revaluation is performed without merely perform- 
ing uniform sampling. 

A further object of this invention is to provide an example of this. 

According to the image processing apparatus of this invention in the aforesaid feature amount weighting reval- 
uation unit, the aforesaid weighting is made to vary by a correspondence relation determined by the position of picture 
elements in the image. 

In the case of a photograph, the person is usually in the center. Therefore, by weightin g the central part of the 
image more heavily after uniformly sampling feature amounts from the whole image, the feature amounts sampled 
from picture elements relating to the person are evaluated to be larger. 

When for example according to the invention thus comprised, the weighting of the central part of the image is 
heavier and the weighting of the surroundings is lighter, in the feature amount weighting revaluation unit, the position 
of picture elements in the image is determined, and a reevaluation is made using a weighting which varies according 
to this position. 

Hence according to this invention, as the weighting is determined according to the position of picture elements, 
the computation is relatively simple. 

The weighting technique is of course not limited to this method, and a further object of this invention is to provide 
20 other examples. 

According to the image processing apparatus provided by this invention, in the aforesaid feature weighting reeval- 
uation unit, the image variation amount is found, and a heavier weighting is given to parts where the image variation 
amount is larger 

In the invention thus comprised, the image variation amount is found before performing reevaluation in the feature 
25 amount weighting reevaluation unit. The image variation amount is also known as imago sharpness, and as tho outline 
is sharper the better the focus, the variation is large where the image is in focus. On a photograph, the part which is 
in focus is the subject, and the part which is not in focus is considered to be the background. Therefore, places where 
there is a large image variation are considered to correspond to the subject. In the feature amount weighting reevalu- 
ation unit, the same result as sampling a large number of feature amounts is obtained by applying heavy weighting to 
30 parts where there is a large image variation. 

According to this invention, as the weighting is varied depending on image sharpness, different targets can be 
precisely identified and feature amounts can be sampled for different images. 

As another example of a weighting technique, in the weighting reevaluation unit of the image processing apparatus 
of the invention, the chromaticity of picture elements is found, a number of picture elements is found for which the 
chromaticityjies within the chromaticity range of the target for which it is desired to sample a feature amount, and 
heavier weighting is applied to parts where there are many of these picture elements. 

Hence according to this invention, in the feature amount weighting reevaluation unit, the chromaticity of picture 
elements is found. In image processing, an object can sometimes be specified by a specific chromaticity. For example, 
there is no reason why a person could not be identified by looking for skin color, but it is difficult to specify skin color 
as color data also contain luminance elements. Chromaticity on the other hand represents an absolute proportion of 
a color stimulation value, and it is not controlled by luminance. Therefore an image of a person could be determined 
if the chromaticity was within a specified range that can be taken as indicative of skin color. This reasoning may of 
course also be applied to the green of the trees or the blue of the sky. 

As specified objects can be sorted by chromaticity according to this invention, different targets may be precisely 
sampled depending on their images and feature amounts sampled. 

Therefore in a feature amount weighting reevaluation unit, when the chromaticity found for picture elements is 
within a chromaticity range for a target from which it is intended to sample feature amounts, plural picture elements 
are counted. When the number of picture elements is large, this part of the image is determined to be the target, heavy 
weighting is applied, and a large feature amount is sampled from the target. 

This weighting technique is not necessarily the only alternative, and it is a further object of this invention to provide 
a suitable example of overlapping methods. 

According to the image processing apparatus of this invention, in the aforesaid feature amount weighting reeval- 
uation unit, temporary weightings are applied based on a plurality of factors, and these factors are thon added according 
to their degree of importance to give final weighting coefficients. 

According to the invention as thus comprised, in the feature amount weighting reevaluation unit, temporary weight- 
ing coefficients are found separately based on a plurality of factors, and the weightings are added according to their 
degree of importance so as to reevaluate the sampled feature amounts as final weighting coefficients. Therefore, it 
may occur that even when a large weighting is assigned by one weighting method in the evaluation stage, if the method 
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does not have a large importance, the final weighting which is assigned is not large. Moreover, it may occur that even 
if there is a large difference between weighting methods, image parts which are evaluated to have an averaqe or hiqher 
weighting also have a large final weighting. 

According to this invention, plural weighting techniques are suitably combined so that a suitable feature amount 
evaluation can be performed. 

Therefore if the image processing indicator specifying unit itself comprises plural forms, it is not necessary to 
perform image processing with only one of these forms. 

However even if there are some cases where it is desirable to perform image processing using the feature amount 
of the object, there are other cases where it is desirable to perform image processing using an average feature amount 
for the whole photographic image. For example, when a photograph is taken of a person, the person may not always 
-bethe4tghtest(highligl Ued) p a , I of t he p i cture. There f ore If attent i on is paid only to the person in the picture and contrast 
is increased, the highlighted part of the background will be too white. In this case, a better result would be obtained by 
paying attention to the whole photographic image. 

Hence when image processing is performed, it is still necessary to select an optimum feature amount 

It is a further object of this invention to automatically select an optimum feature amount according to imaqe process- 
ing technique. 

In the image processing apparatus according to this invention, photographic image data comprising dot matrix 
picture elements is input, and predetermined image processing is performed. This image processing apparatus com- 
prises an image data acquiring unit for acquiring the aforesaid photographic image data, an image processing indicator 
specifying unit for performing a predetermined summation processing on picture elernenls based on this acquired 
image data, and specifying an image processing indicator, and a processing unit for determining image processing 
contents based on the specified indicator, and performing image processing. The aforesaid image processing indicator 
specifying unit comprises an evaluation unit for obtaining a feature amount by inputting the aforesaid photographic 
data, summing the image data for all picture elements, and obtaining a feature amount according to plural predeter- 
mined evaluation criteria. In the aforesaid processing unit, the imago data can be converted by plural techniques and 
the feature amounts obtained in the aforesaid evaluation unit used according to the particular technique 

According to this invention, image data from a photographic image comprising dot matrix picture elements is input 
in this manner, and a feature amount is obtained according to plural evaluation criteria by summing image data for 
picture elements in the evaluation unit In the processing unit, in converting the image data by plural techniques the 
feature amounts obtained in the evaluation unit accordingto each technique are then used to convert the data depend- 
ing on the technique. 

Specifically, although there are cases where the image data is best converted using a feature amount centered 
on the object such as in light/dark correction, there are other cases where image data is better converted using a 
feature amount centered on the whole image such as when contrast is increased, and the image data conversion may 
be performed by suitably selecting these plural feature amounts. 

Accortling to this invention, when feature amounts are obtained according to plural evaluation criteria and image 
processing is performed by plural methods, the feature amounts used depend on the method, so image processing 
may be performed based on an optimum evaluation criterion. 

When image data is converted, the feature amounts should be such that they can be used to identify the features 
of the image, and there is no need to specify the type of image. For example this also includes indicators such as 
luminance histograms which identify whether the image is to be considered as light or dark, there being no need to 
obtain the identification result that the image is light or dark. Apart from lightness, the indicator may of course specify 
whether or not the image is sharp, or it may be an indicator to identify vividness. 

There is also no particular limitation on the way in which the evaluation unit and processing unit are applied. For 
example, assuming that plural image processing methods are used, plural feature amounts may be obtained and stored 
according to plural evaluation criteria, and the image data converted by suitably selecting the feature amount in the 
processing unit as necessary. As another example, image data for picture elements may be summed by a predeter- 
mined criterion in the evaluation unit so as to obtain a suitable feature amount on each occasion that image processing 
is performed in the aforesaid processing unit 

It will of course be understood that these plural image processing methods based on feature amounts obtained 
by plural different evaluation criteria, may also be applied not only to an actual device but also to a system both of 
which are then a valid form of the invention. When the image processing methods are implemented by software as 
specific examples of the concept of the invention, thore naturally exist media on which the software is recorded which 
then offer all the attendant advantages thereof. 

The evaluation criterion used to obtain feature amounts in the aforesaid evaluation unit will depend on the image 
processing that is to be performed, and while there are some cases where it is desirable to concentrate on the object 
for image processing, there are some cases where it is not as described above. 

It is a further object of this invention to provide an example of the former case. 
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In the image processing apparatus according to this invention, the aforesaid evaluation unit comprises an evalu- 
ation unit wherein an object in a photographic image is sampled, and image data for picture elements of this object is 
summed to obtain a feature amount, and in the aforesaid processing unit, in one processing method, the feature amount 
obtained from object picture elements is used when the feature amount for the central part of the image data is used 

According to the invention thus comprised, when image data is converted based on the feature amount of the 
central part of the image data in the aforesaid processing unit, the object in the photographic image is sampled in the 
aforesaid evaluation unit, and the feature amount is obtained by summing image data for object picture elements 
according to predetermined criteria. 

Herein, the central part of the image data has the following meaning. For example, when it is determined whether 
a given photograph is light or dark, it is easily appreciated that the determination can conveniently be based on the 
-fntermeaiaie aensity or the image. I his intermediate density may also be referred to as a median in a luminance 
distnbution, i.e. the center of the luminance distribution, and in this sense it is referred to as the central part of the 
image data. Then, if there is an object in the photographic image, it may be said that there is a definite necessity to 
perform light/dark correction in line with the lightness of this object.. 

This invention is suitable for the case when image processing is performed based on the feature amount of the 
central part of the image data. 

Any of the aforesaid techniques may be applied as the basic technique for sampling the object As an example 
in the aforesaid evaluation unit, picture elements for which there is a large variation of image data between adjacent 
picture elements are sampled as the object. When picture elements are aligned at a fixed interval apart as in the case 
of a dot matrix image, the difference of image data between adjacent picture elements is proportional lo a first order 
differential. This difference may be determined as the image variation amount. In this case, the difference may be 
regarded as the magnitude of a vector, and the vectors constructed taking account of adjacent directions. If this is done 
it is sufficient to determine the difference of image data for adjacent picture elements, computing is easy, and the 
processing for determining the object is reduced. 

As another example, in the aforosaid evaluation unit, picture elements for which the chromaticity is within a pro- 
determined range may be sampled as the object. In this case, in the aforesaid evaluation unit, the chromaticity of 
picture elements is found. The chromaticity represents an absolute proportion of a color stimulation value, and it is not 
affected by lightness. Therefore the object in the image can be separated by possible range of chromaticity. For ex- 
ample, there is the chromaticity range for skin color, or the'ehromaticity range for the green of the trees. As this can 
be said for chromaticity. in the aforesaid evaluation unit, picture elements for which the chromaticity lies within a pre- 
determined range are sampled as the object. In this way. an object can be determined by its chromaticity. and the 
object may be sampled without depending on the lightness or darkness of the object. 

On the other hand as an example of image processing not concerned only with the object, the aforesaid evaluation 
unit comprises an evaluation criterion wherein picture elements of the aforesaid image data are uniformly sampled and 
summed soas to obtain a feature amount, and in the aforesaid processing unit, in one processing method, the feature 
amount obtained by the aforesaid uniform sampling is used when an average feature amount of the photographic 
image is used. In this case, when image data is converted based on the average feature amount in the aforesaid 
processing unit, the feature amount is obtained by uniformly sampling picture elements of the image data according 
to predetermined evaluation criteria. Of course, the summation may be performed on all picture elements of the pho- 
tographic image, but it may be said that is no advantage as the processing amount increases. Hence, it is convenient 
to perform image processing based on the average feature amount of the photographic image, e.g. saturation correc- 
tion. 

As another example of image processing which is not concerned only with the object, the aforesaid evaluation unit 
comprises an evaluation criterion wherein picture elements of the aforesaid image data are uniformly sampled and 
summed to obtain a feature amount, and in the aforesaid processing unit, in one image processing method, the feature 
amount obtained by uniform sampling is used when the edges of a feature amount distribution of the photographic 
image are used. In this case, in the aforesaid processing unit, it is assumed that the ends of the feature amount dis- 
tribution obtained in the aforesaid evaluation unit are used. For example, to increase the contrast, image processing 
is performed to find the luminance distribution, and the edges of this luminance distribution are widened, but if the 
luminance distribution of the object were used in this case, other highlighted parts appear white. Therefore in this case, 
in the aforesaid evaluation unit, picture elements of image data are uniformly sampled according to predetermined 
cntena, and summed to obtain the feature amount. This is suitable for image processing using the edges of a feature 
amount distribution in an actual photographic imago, e.g. for increasing contrast. 

In the above, a continuous sequence of processes is performed comprising predetermined analysis of the image 
and image processing by specifying image processing indicators, but the analysis result itself is also useful. 

It is a further object of this invention to provide an image evaluation device wherein it is easier to use an image 
evaluation result which is an analysis result done. 

In the image evaluation device offered by this invention, photographic image data comprising dot matrix picture 
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elements is input, the .mage data for all picture elements is summed according to predetermined criteria and is the 
image evaluation dev.ce which is based summation result, and evaluate image, and the image is evaluated based on 
the summation results. There are plural evaluation criteria for the aforesaid summation results, and the evaluation 
results are combined with a predetermined weighting based on these evaluation criteria. 

According to the invention as thus comprised, the evaluation method assumes that photographic image data com- 
prising dot matrix picture elements is input, the image data is summed for picture elements, and the image is evaluated 
based on the summation results. Herein, there are plural evaluation criteria for these summation results, and the eval- 
uation results are combined with a predetermined weighting based on the evaluation criteria 

In other words, although some evaluation criteria are suitable for evaluating images where a sharp image is the 
object such as in the case of portrait, other criteria are suitable for evaluating images where the background is the 
important oUjhU. A general avalualion may be made- by suitably combining plura l evaluation criteria in parallel and ~ 
varymg the weightings. 

As described hereabove. as th is invention gives a general evaluation by varying the weightings of plural evaluation 
criteria, it provides an image evaluating device which can be flexibly adapted to image feature determination 

Naturally, the concept of this invention for image evaluation by the above techniques comprises many forms Spe- 
cifically, it comprises hardware and software, various modifications being possible as may be convenient When the 
concept of the invention is implemented by image processing software, there naturally exist recording media on which 
the software is recorded which can be used to perform the function of the invention. Moreover, these image evaluating 
techniques may be applied to an image evaluating device and its system running on software media 

Various techniques may be employed lo achieve the same object in applying plural evaluation criteria to summation 
results. For example, all picture elements may be summed by weighting with different evaluation criteria, but it will be 
appreciated a large amount of processing is involved when the summation is applied to all picture elements Hence 
as described above, the picture elements are first sampled based on plural evaluation criteria, summed, and the sum- 
mation results combined with a predetermined weighting. In this case, the image data are sampled prior to summation 
plural criteria are used by varying the criteria applied to the sampling, and tho weighting of the summation results is 
then adjusted before combination. An evaluation can therefore be made with different weightings on the results based 
on plural evaluation criteria. Due to this sampling of image data, plural evaluation criteria may be employed according 
to the sampling method. s 
As one evaluation criterion, the data may of course be sampled uniformly and summed. In this case the image 
data is uniformly thinned and the whole image is considered, which makes this a suitable criterion for determining 
scenic photographs, etc. In this way an optimum criterion can be used while reducing the processing amount 

As an example of a criterion which can be applied whether or not sampling is used, is the evaluation of picture 
elements which have a large variation relative to adjacent picture elements with a heavier weighting In this case the 
image variation of picture elements from adjacent elements is detected, and clear image parts with a large variation 
are g.venaheavier weighting in the summation. If this is done, as image parts with a large variation are often parts of 
the photograph which are clearly in focus, an image evaluation which weights important parts of the image can be 
performed. 

This criterion places more emphasis on the sharp parts of the image, and it is therefore naturally suitable for the 
determination of human images. Herein, image parts with a large variation may be evaluated either by introducing 
weighting as the picture elements are summed, or by summing only picture elements with a large variation The weight- 
ing used in the evaluation is not necessarily fixed, but may also be allowed to vary according to the criterion In this 
case : by varying the weightingfor different evaluation criteria, an overall evaluation result forthe image can be deduced 
Moreover various approaches are possible, e.g. weightings may be varied individually so as to generate plural com- 
binatons which are then selected. In this way. by modifying the weighting for plural criteria, a more flexible evaluation 
can be made. 

Instead of an operator varying the weighting, this can be done based on the Image data itself As an example of 
this, the weighting of the evaluation results may be varied based on the criteria. In this case, results are obtained 
according to various criteria, and the weighting is modified in view of the suitability of the criteria in the light of the 
results. As the results are used to vary the weighting, the work involved in the evaluation is less. 

various techniques may also be used to modify the weighting of the criteria using the results For example if it is 
determined whether or not the image data for picture elements should be sampled according to one criterion the 
number of these picture elements may be taken as a criterion and the weighting increased when the number of picture 
elements is large. 
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Brief Description of the Invention 

Fig. 1 is a block diagram of an image processing system in which an image processing apparatus according to 
one embodiment of this invention is applied. 
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Fig. 2 is a block diagram of the actual hardware of the image processing apparatus. 

Fig. 3 is a block diagram showing another application example of the image processing apparatus of this invention 
Fig. 4 is a block diagram showing another application example of the image processing apparatus of this invention. 
Fig. 5 is a flowchart showing a front stage of main processing in an image processing apparatus according to this 
invention. 

Fig. 6 is an illustration showing a case where an image variation amount is expressed by component values in 
orthogonal coordinates. 

Fig. 7 is an illustration showing a case where an image variation amount is expressed by a differential value in 
adjacent picture elements in a vertical axis direction and a horizontal axis direction. 

Fig- B is an illustration showing a case when an image variation is calculated between ad jacent pi cture elements. 

Fig. 9 is a diagram showing a region wherein a threshold value is varied. 
Fig. 10 is a flowchart for automatic division of regions. 
Fig. 11 is a diagram showing a region setting. 

Fig. 12 is a diagram showing this region setting in a modified example. 
Fig. 1 3 is a flowchart showing the latter half of main processing. 

Fig. 14 is a diagram showing the edges obtained by luminance distribution edge processing and edge processing. 
Fig. 15 is a diagram showing widening of a luminance distribution and a reproducible luminance range. 
Fig. 16 is a diagram showing a conversion table for widening the luminance distribution. 
Fig. 17 is a diagram showing the general concept of brightening by y correction. 
Fig. 18 is a diagram showing the general concept of darkening by y correction. 
Fig. 19 is a diagram showing a correspondence relation for luminance modified by y correction. 
Fig. 20 is a flowchart showing a case when saturation is emphasized in the latter part of main processing. 
Fig. 21 is a schematic view of a summation state of a saturation distribution. 
Fig, 22 is a diagram showing a relation between a saturation A and saturation emphasis index S. 
Fig. 23 is a flowchart showing a caso when edges are emphasized in tho latter part of main processing. 
Fig. 24 is a diagram showing the magnitude of image data and a state where image data to be processed is 
displaced. 

Fig. 25 is a diagram showing a 5x5 picture element unsharp mask. 

Fig. 26 is a block diagram of an image processing system in which an image processing apparatus according to 
30 one embodiment of this invention is applied. 

Fig. 27 is a flowchart showing image processing in the image processing apparatus according to this invention. 
Fig. 28 is a diagram showing a sampling frequency. 
Fig. 29 is a diagram showing a picture element sampling number. 

Fig. 30 is a diagram showing a relation between an image to be converted and picture elements for sampling. 
Fi 9- 31 diagram showing a block arrangement resulting from image division. 
Fig. 32 is a diagram showing a block luminance distribution. 
Fig. 33 is a diagram showing an example of block weighting. 
Fig. 34 is a diagram showing another example of block weighting. 

Fig. 35 is a flowchart showing a case when a feature amount is reevaluated based on an edginess amount. 
Fig. 36 is a diagram showing a relation between a main picture element and edge picture elements for determining 
the edginess amount. 

Fig. 37 is a diagram showing an example of a filter for computing the edginess amount. 
Fig. 38 is a flowchart showing a case when a feature amount is reevaluated based on an edginess amount. 
Fig. 39 is an example of a photographic image. 

Fig. 40 is a diagram of a luminance distribution of a photographic image photographed at night. 
Fig. 41 is a block diagram of an image processing system in which an image processing apparatus according to 
one embodiment of this invention is applied. 

Fig. 42 is a flowchart showing a sampling part in an image processing apparatus according to this invention. 
Fig. 43 is a flowchart showing a feature amount acquiring part and an image processing part. 
Fig. 44 is a block diagram of an image processing system in which an image processing apparatus according to 
one embodiment of this invention is applied. 

Fig. 45 is a flowchart showing an image processing part in an image processing apparatus according to this in- 
vention. 
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Fig. 46 is a block diagram showing an image evaluation option input screen. 

Fig. 47 is a diagram showing how individual sampling results are converted to weightings and combined. 
Fig. 48 is a flowchart showing the latter stage of image evaluation and an image processing part. 
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Description of Special Embodiment 

Some preferred embodiments of this invention will now be described. 
Embodiment 1 

First, a description will be given of one form of image processor comprising an image processing indicator specifying 
unit which comprises an object determining unit, and automatically determines an object as picture elements having 
a large variation. Conventionally, this determination had to be performed by a human operator 
.„ „ri 9 ll! h< ^ WS a bl ° c k di a 9 ram ot an ima 9e processing system to which an image processing apparatus according 
lu one embodiment of this invention is applied. Mg. 2 snows an example of the actual hardware construction by a 
schematic block diagram. 

In Fig. 1, an image reader 10 outputs photographic image data which represented photographs as dot matrix 
picture elements to an image processing apparatus 20A. The image processing apparatus 20A determines the contents 
and extent of image processing, and then performs the processing. The image processing apparatus 20A outputs the 
processed image data to the image output apparatus 30, and the image output apparatus 30 outputs the processed 
image as dot matrix picture elements. From the image data output by the image processing apparatus 20A, an image 
variation amount is found for each picture element, and picture elements having a large variation amount are determined 
to be those of the object. The content and extent of image processing are determined, and image processing is per- 
formed in line with this object image data. Therefore, the image processing apparatus 20A comprises an object deter- 
mining unit which finds an image variation amount in each picture element and determines picture elements having a 
large variation amount as the object, and processing unit which determines the content and extent of image processing 
in line with object image data. 

A scanner 1 1 in Fig. 2 and digital still camera 1 2 or video camera 1 4 correspond to a specific example of the image 
reader 10, the computer system corresponds to a specific example of the imago processing apparatus 20A comprising 
a computer 21, hard disk 22, keyboard 23. CD-ROM drive 24, floppy disk drive 25 and modem 26, and the printer 31 
and display 32 correspond to specific examples of the image output apparatus 30. In case of this embodiment, the 
object is found to perform appropriate image processing, so photographic data such as photographs are suitable as 
image data A modem 26 is connected to the public telecommunication line, and to an external network via the public 
30 telecommunication line through which software and data can be downloaded. 

According to this embodiment, the scanner 11 and digital still camera 12 which function as the image reader 10 
output RGB (red, green, blue) gradation data The printer 31 which is the image output apparatus 30 requires input of 
CMY (cyan, magenta, yellow) or CMYK (to which black is added) as gradation data, and the display 32 requires RGB 
gradation data as input. Also, an operating system 21a runs on the computer 21 . A printer driver 21 b corresponding 
to the printer31 and display driver 21c for the display 32 are built in. Processing is controlled by the operating system 
21a, and an image processing application 21 d performs predetermined image processing together with the printer 
driver 21b and display driver 21c when necessary. Therefore, the specific role of this computer 21 which functions as 
the image processing apparatus 20A is to input RGB gradation data, generate RGB gradation data for optimum image 
processing, display the data on the display 32 via the display driver 21c, convert the data to CMY (or CMYK) binary 
data via the printer driver 21 b, and print it on the printer 31 . 

In this way, according to this embodiment of the invention, a computer is interposed between the inputoutput 
apparatus to perform image processing, but a computer is not absolutely necessary if the system is capable of per- 
forming various types of image processing on image data. For example the system may be such that the image process- 
ing apparatus which determines the object and performs image processing is built into the digital camera 1 2a as shown 
in Fig. 3. The image is displayed on a display 32a and printed by a printer 31a using converted image data. Alternatively, 
the object is determined and image processing is performed automatically from image data input via a scanner lib 
and digital still camera 12b or modem 26b. as shown in Fig. 4. 

The aforesaid determination of the object and image processing are performed by an image processing program 
corresponding to a flowchart shown by Fig. 5 which is built into the computer 21. In the flowchart shown in the figure, 
it is determined whether or not the image is that of the object. 

According to this invention, picture elements for which the image is sharp are determined to be those of the object 
based on the experimental fact that the image is sharper for the object than for other parts. When image data comprises 
dot matrix picture elements, gradation data is displayed showing RGB luminance for each picture element, and a 
difference amount between data for adjacent picture elements becomes large at the edge of the image. This difference 
amount is a luminance gradient, and is referred to as edginess. In a step SA110, the edginess of each picture element 
is determined. When the XY orthogonal coordinate system is considered in Fig. 6, vectors of the image variation amount 
may be computed if the X-axis direction component and Y axis direction component are found respectively. For a digital 
image comprising dot matrix picture elements, assume that there are adjacent picture elements in the vertical axis 



35 



40 



45 



SO 



ss 



11 



EP 0 891 075 A2 



direction and horizontal axis direction as shown in figure 7, and assume that the luminance is expressed as f (x y) In 
this case f (x. y) is R(x. y). G(x, y), B(x. y) which is the luminance of each of the colors ~- ^B, or it may be a total luminance 
Y(x. y). Strictly speaking, the relation between R(x. y). G (x, y), B (x. y) which is tr_ luminance of each of the colors 
RGB and total luminance Y(x. y). cannot be converted without referring to color conversion charts but a simple cor- 
respondence relation can be utilized as described hereafter. As shown in figure 7, a difference amount value fx in the 
X direction and a difference amount value fy in the Y direction may be written: 



fx = f (x+1,y)-f (x,y) (1) 
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fy = f (x.y+1)-f (x,y) {2) 
Therefo-c the magnitude of the vector Ig (X, y)l having these differences as components may be written as: 

r 

lg(x,y)l = (fx"2 + fy"2)~(1/2) (3) 

Edginess is of course represented by lg(x, y)l. The picture elements are actually arranged in length and breadth as a 
grid shown in Fig 8 the.e being eight picture elements in the center. Therefore, expressing a difference amount of 
image data between adjacent picture elements as a vector, the sum of this vector may be taken as the image variation 
amount. 

As the edginess may be found for each picture element in this way, picture elements having a large edginess when 
compared with a threshold value may be dctorminod to bo object picture elements. However from actual experience 
the object is often situated in the center of the figure. This fact provides proof that the use of an arrangement where a 
large number of picture elements are sampled from the center area for image processing, gives satisfactory results 

For this reason, the threshold values Th1, Th2, Th3 used for comparison in each part of the center of the image 
are arranged to be different as shown in Fig. 9. In this example, of course, the relation 

Th1 < Th2 < Th3 (4) 

holds. The threshold value is lower the nearer the center, and this area is determined to be the object even if the 
edginess is relatively low. 

As theUueshold value varies as shown in the figure, the area is divided uniformly into three equal parts in the 
horizontal and vertical directions from the center of the image. In a step SA120, the threshold value used for comparison 
is determined based on the area in which the picture element used for edginess determination is located The threshold 
value is compared with the edginess in a step SA1 30, and it is determined whether or not the variation amount is large 
As a result of this comparison, if the edginess is large, it is determined that this picture element is a picture element of 
the object, and image data of the picture element is stored in a work area in a step SA140. The work area may be a 
RAM in the computer 21 , or the hard disk 22. 

The above processing is performed for each picture element of image data. In a step SA150\ the picture element 
to be processed is displaced, and processing is repeated until it is determined that processing has been completed in 
a step SAl 60. 

In the embodiment described above, the area division for modifying the threshold value was always based on the 
central pan of the image, but the area division may also be varied based on the edginess distribution. Fig. 10 is a 
flowchart for suitably varying the area division, and Fig. 11 shows the areas so divided. 

In this case also, subsequent processing is performed on each picture element while moving the picture element 
to be processed in the same way as above. After the aforesaid edginess was determined in a step SA21 0, it is summed 
in the horizontal axis direction in a step SA220, and summed in the vertical axis direction in a step SA230. Picture 
elements to be processed are displaced in a step S A240, and the process loops until it is determined in a step SA250 
that processing for all picture elements is complote. 

After summation is completed for the horizontal axis direction and vertical axis direction, a maximum distribution 
position on the horizontal axis is determined in a step SA260, and a maximum distribution position on the vertical axis 
is determined in a step SA270. As shown in Fig. 1 1 , a high edginess part along the horizontal and vertical axis direction 
is regarded as the center of the image, the area being divided as follows. 

The distance to an end from the center is divided in half in the vertical direction and the horizontal direction. The 
threshold value is Th1 for the inside, the remaining distance being divided in half with a threshold value Th2 for the 
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inner area and a threshold value Th3 for the outer area. In a step SA280. by dividing the area as described above 
comparison criteria are determined, and in a step SA290, object picture elements are determined by perform sampling 
based on edgmess by the same processing as that of the steps SA11O-SA160 mentioned above according to the 
correspondence between this area and threshold value. 

In this example, after finding the center area, the area was divided into two equal parts in the directions of both 
the horizontal axis and vertical axis, but the area may also be divided in other ways, e.g. based on the edginess 
distribution. The actual area division may be suitably varied. 

For example, in the example stated above, summation in the horizontal axis direction and vertical axis direction 
were performed in picture element units, but the image may be divided into a relatively larger grid as shown in Fig 12 
The summa tion may then be made in these grid units, the position of the maximum distribution determined and area 
division performed. ~ — — — 

If object picture elements can be sampled in this way. the optimum image processing can be determined and 
performed based on image data for these picture elements. Fig. 1 3 is a flowchart showing increase of contrast and 
lightness compensation as an example. 

In the basic technique to increase contrast according to this embodiment; a luminance distribution is found based 
on object image data, and if this luminance distribution uses only part of the original gradation (255 qradations) the 
distribution is expanded. 

Therefore, a histogram of luminance distribution is generated in a step SA310. and an expansion width is deter- 
mined in a step SA320. When the expansion width is determined, both ends of the luminance distribution are found 
A luminance distribution of a photographic image is generally like a hump as shown in figure 14. The distribution may 
of course have various positions and shapes. The width of the luminance distribution is determined by where the two 
ends are located, but the points where the distribution number is "0" where the distribution slopes away cannot be 
taken as the ends. There is a case where the distribution number varies in the vicinity of "O" in the lower sloping part. 
This is because from a statistical viewpoint, it changes without limit as it approaches "0\ 

Therefore, the two ends of the distribution are taken to bo a part in the distribution shifted somewhere towards tho 
insfde by a certain distribution fraction from the brightest side and the least bright side. In the area in this embodiment 
this distribution fraction is set to 0.5%. but the fraction may be modified as deemed appropriate. In this way, white spots 
and black spots due to noise can also be ignored by cutting the upper and lower ends by a certain distribution fraction 
Specifically, if such processing is not performed and there are white spots or black spots, these become the two ends 
of the luminance distribution. In a 255 gradation luminance value distribution, the lower end is usually "CT and the upper 
limit is "255-. but the above problem is avoided by considering the end of the distribution to be a point situated at 0 5% 
from the end in terms of picture elements. 

In the actual processing. 0.5% of the number of picture elements sampled as the object is computed, the distribution 
numbers are summed towards the inside in sequence from the luminance value at the upper end and the luminance 
value at jhe lower end in a reproducible luminance distribution, and the luminance value corresponding to 0 5% is 
determined? Hereafter, the upper limit will be referred to as ymax and the lower limit as ymin. 

When the reproducible range of luminance is "0" - '255\ the luminance Y converted from the luminance y before 
conversion, and the maximum value ymax and minimum value ymin of the luminance distribution is given bv the 
following equations: 

Y = ay*b (5) 

where 

a - 255/(ymax-ymin) ^ 



so b = -a. ymin or 255-a.ymax (7) 

In the above equation, when Y<0, Y is set equal to 0. and when Y>255. Y is set equal to 255. a is a slope and b 
is an offset. According to this conversion equation, a luminance distribution having a certain narrow width can be 
enlarged to a reproducible range, as shown in Fig. 15. However, when the reproducible range was increased to the 
55 maximum to expand the luminance distribution, highlighted areas are white and high shadow areas come out black 
To prevent this, according to this embodiment, the reproducible range is limited, i.e. a luminance value of *5" is left as 
a range which is not expanded at the upper and lower ends of the reproducible range. As a result, the parameters of 
the conversion equation are given by the following equations: 
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A = 245/(ymax-ymin) (g) 



B = 5- a/ymin or 250-a/ymax rgj 



In this case, in the ranges y<ymin and y>ymax, conversion is not performed. 

However, if this expansion factor (corresponding to a) is applied, a very large expansion factor may be obtained. 
For example at dusk, although the width of contrast from the brightest to the darkest part is naturally narrow, if the 
contrast of such an image were considerably increased, it would appear to be converted to a daytime image. As such 
a conversion is not desired, a limit is imposed on the increase factor so that it is equal to or greater than 1.5 (-2) . Due 
to this, dusk correctly appears as dusk. In this case, processing is performed so that the center position of the luminance 
distribution does not vary. 

However, in luminance conversion, it is unreasonable to perform the aforesaid conversion (Y = ay+b) on every 
occasion. This is because the luminance y can only lie within the range "0" to "255-, and the luminance Y after conversion 
can be found for all possible values of y beforehand. These can therefore be stored in a table such as is shown in Fig. 16. 

This conversion table corresponds to the expansion width determination processing of the step SA320, and it 
allows image data to be converted. However, as it is very useful not only to emphasize contrast by increasing the 
luminance range but also to adjust luminance at the same time, the luminance of the image is determined in a step 
20 SA330 and a correction parameter is generated. 

For example, the hump of the luminance distribution may be displaced to the side which is brighter overall as 
shown by the dotted line in Fig. 17 when the hump is nearer dark as shown by the solid line in the figure. Conversely, 
the hump of the luminance distribution may be displaced to the side which is darker overall as shown by the dotted 
line in Fig. 18 when the hump is nearer bright on the whole as shown by the solid line in the figure. 
25 By performing various experiments, according to this embodiment, a Median ymed in the luminance distribution 

is found. When the median ymed is less than '85*. the image is determined to be dark, and is lightened by ay correction 
corresponding to the following y value: 

30 y = ymed/85 ( 10 ) 



or 



35 ^ y = (ymed/85) * * (1/2) 

In this case even if y < 0.7, y is set equal to 0.7. If such a limit is not provided, a night scene appears as if it is in 
daylight. If the image is made too light, it becomes too white and contrast tends to be too low, hence it is preferable to 
perform processing such as emphasis in conjunction with saturation. 
40 On the other hand, when the median ymed is greater than "123", the image is determ ined to be a light image and 

is darkened by a y correction corresponding to the following y value: 

y - ymed/ 128 (12) 

or 

y=(ymed/128)**(1/2) . _ . (13). 



In this case oven if y > 1 .3 : y is set equal to 1 .3 so that the image does not become too dark. 

This y correction may bo applied to the luminance distribution before conversion, or to the luminance distribution 
after conversion. Correspondence relations for y correction are shown in Fig. 1 9. When y < 1 , the curve bulges upwards, 
whereas when y >1 , the curve bulges downwards. Of course, the result of this y may also be reflected in the table 
ss shown in Fig. 16, and the same correction may be applied to table data. 

Finally, in a step SA340, it is determined whether or not contrast correction and lightness compensation are nec- 
essary, in this determination, the aforesaid expansion factor (a) and y value are compared with suitable threshold 
values, and when the expansion factor is large and they value exceeds a predetermined range, it is determined that 
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such correction is necessary. If it is determined to be necessary, conversion of image data is performed Specifically 
the need for image processing and its extent are assessed in the steps SA310 - SA340. and the required imaae 
processing is performed in the a SA350. The processing unit to accomplish this comprises hardware and software 

Conversion on the basis of equation (5) is performed when it is determined that image processing is necessary 
This equation may also be applied to correspondence relations between RGB component values. The component 
values after conversion (R : G, B) relative to the component values before conversion (RO, GO. BO) may be found from 



R = a ■ RO+b 



(14) 



G = a - GO+b 



B = a • BO+b 



(15) 
(16) 



nrS' th fn^! c C . 0,T1P0 1 " en, ValUSS (R0, G °' ^ (R G " B) hava the same ran 9 e when the finance y Y has the 
gradation 0 255 . so the aforesaid conversion table of luminance y, Y may be used without modification 

em « ° re ' *™ conversion lab,e corresponding to equations (14) - (16) is referred to for the image data (RO GO 
BO) for all P ,cture elements in a step SA350. and the process for obtaining the image data (R. G. B) after conversion 
is repeated. 

In this processing unit, the determination is performed only for contrast correction and lightness compensation 
out specific examples of image processing are not limited to this. 

Fig. 20 shows a flowchart for performing image processing for saturation emphasis 

First, if the object and determined picture element data have saturation as a component element, a distribution 
may be found using saturation values. However, as the data comprises only RGB component values, saturation values 
cannot be obtained unless they are converted to a color specification space which comprises direct component values? 
For example n a Luv space which is a colorimetric system, the L axis represents luminance (lightness) and hues are 
represented by the U axis and V axis. Herein, as the distance from the intersection point of the U axis and V axis shows 
saturation, the saturation is effectively (U**2+V**2)**(1/2). 

Such a color conversion between different color specification spaces requires an interpolation to be performed 
while refernng to a color conversion table which stores correspondence relationships, and the computation amount is 
enormous. In view of this, according to this embodiment, standard RGB gradation data is utilized directly as image 
data, and saturation substitute values X are found as follows: 

X=IG+B-2XRI (17) 

Actually, the saturation is "0- when R = G = B, and is a maximum value either for any of the single colors RGB or 
for a nurture of two colors in a predetermined proportion. Due to this, it is possible to appropriately represent saturation 
directly. For yellow which is a mixture of the color red with green and blue, from the simple equation (17). the maximum 
saturation value is obtained, and when the components are equal, this is -0". For green or blue alone about half the 
maximum value is attained. Of course, substitutions may be made using the equations: 

X' = IR+B-2XGI (18) 

X"=IG+R-2XBI (19) 

In a stop SA41 0, a histogram distribution is found for the saturation substitution value X. In equation (17) saturation 
is distributed in the range of minimum value "0' - maximum "511". and the distribution obtained is approximately as 
shown in Fig. 21. In a next step SA420. based on the summed saturation distribution, a saturation index is determined 
or this image According to this embodiment, a range occupied by the upper 1 6% of distribution number is found within 
the number of picture elements determined to be the object. Assuming that the lowest saturation "A' in this ranqe 
represents the saturation of the image, a saturation emphasis index S is determined based on the following equation 
In other words, it is assumed that: 
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If A<92, S = -Ax(10/92)+50 (2 0) 
If 92 ^A<184, S = -AX(10746)+60 (21) 
If 184 ^A<230, S= -AX(10/23)+100 (22 ) 
If 200 f?- A, S = 0 
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Fig. 22 shows a relation between this saturation "A" and the saturation emphasis index S. As shown in the figure 
in the range between the maximum value "50" - minimum value "0", the saturation index S gradually varies so that it 
is large when the saturation A is small, and small when the saturation A is large. 

When saturation is emphasized based on the saturation emphasis index S, it the image data is provided with 
saturation parameters as stated above, the parameters may be converted. When an RGB color specification space is 
adopted, the data must first be converted into the Luv system which is a standard color system, and moved in a radial 
direction with.n the Luv space. However, this means that RGB image data must first be converted into image data in 
Luv space, and then returned to RGB after saturation emphasis which involved an enormous amount of computation 
Therefore, RGB gradation data are used without modification for saturation emphasis. 

When the components are component values of hue components which are in a schematic pair relatbn as in the 
case of the RGB color specification space, the color is grey and there is no saturation if R = G = B Therefore if the 
component which has the minimum value in RGB is considered moroly to havo a reduced saturation without having 
any effect on the hue of the picture elements, the minimum of each component may be subtracted from all the com- 
ponent values, and the saturation emphasized by increasing the value of the difference. 

First, a parameter Sratio which is useful for calculation is found from the aforesaid saturation emphasis index S 
by the equation: 

Sratio = (S+1 00)/100 (2 4) 

In this case the saturation emphasis parameter Sratio = 1 when the saturation emphasis index S =O t and saturation 
is not emphasized. Next; assuming that the value of the blue (B) component in the components (R, G, B) of the RGB 
gradation^ita is the minimum, this saturation emphasis parameter Sratio is used to perform the following conversion: 

R' = B+(R-B) X Sratio (25) 
G* = B+(G-B) X Sratio {26 ) 

B' = B (27) 

As a result, two-way color conversions between the RGB color specification space and the Luv space are rendered 
unnecessary, and computing time can be reduced. In this embodiment, as for non-saturation components, the com- 
ponent with the minimum value was simply subtracted from other component values, but other conversion equations 
may be used to subtract non-saturation components. However when only the minimum values are subtracted as in 
equations (25)-(27), there are no multiplications or divisions so the computation is easier. 

When the equations (25)-(27) are used, a good conversion is possible, however in this case when saturation is 
emphasized, luminance also increases so the image becomes lighter overall. Therefore, the conversion is performed 
using a difference value obtained by subtracting an equivalent luminance value from each component value- 
Firstly to find the luminance, as the computation becomes bulky when a color conversion is performed in Luv 
space, the following equation which is used in television for example, is used to find the luminance from RGB The 
luminance Y is given by 
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Y=0.30 R+0.59G+0.11B (28) 
It will be assumed that saturation emphasis is given by 

R 1 ~ R+AR (29) 

io G* = G+AG (3qj 
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B' = B+AB (31) 

These addition/subtraction values AR, AG, AB are found by the following equations based on difference amount 
values: 

AR = (R-Y) x Sratio ( 32) 



AG = (G-Y) x Sratio 



(33) 



25 AB - (B-Y) X Sratio (34) 

As a result, the conversion can be performed by 



30 R' = R+(R-Y) x Sratio 



(35) 



G' = G+(G-Y) x Sratio (36) 

. B'= B+(B-Y) x Sratio ( 37 ) 

Conservation of luminance is clear from the following equations: 



Y'= Y+AY 

= 0.30 ^R-i-O. 59^G+0.11Z\B 

= Sratio { ( 0 . 30R+0 . 59G + 0 . 1 IB) -Y} 

= 0 . . . (39) 



(38) 



Also, when the input is grey (R = G = B), the luminance Y = R = G = B, the additiorvsubtraction values AR = AG = AB 
- 0 and there is no color in the achromaticity. If equations (35) - (37) are utilized, the luminance is stored, and the 
image does not become lighter overall even if saturation is emphasized. 

If the saturation emphasis index Sratio is found as described above, it is compared with a predetermined threshold 
ss value in a step SA430, and it is determined whether the image requires saturation emphasis. If it is necessary, the 
image data for all picture elements is then converted based on equations (35)-(37) in a step SA440. 

Therefore, in the steps SA410-SA430, the need for saturation emphasis processing and its extent is determined 
and when it is deemed necessary in the step SA430, saturation emphasis processing is performed. A processing unit 
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therefore comprises hardware and software to accomplish these functions. 

To determine the content and extent of image processing based on object picture elements, edge emphasis 
processing may also be used. Fig. 23 shows a flowchart of this edge emphasis processing. As object picture elements 
are selected, the edginess is averaged for object picture elements by dividing the integrated edginess by the number 
of picture elements in a step SA510. If the number of picture elements is E (I) Pix. the sharpness degree SL of the 
ob|ect image may be computed by 

SL = X l£r<x.,y>|/E (I) pix ... (40). 



In this case, the degree of sharpness is lower (the image appears more blurred) the lower the SL value of the 
image, and the degree of sharpness is higher (the image appears clearer) the higher the SL value of the image 

On the other hand because the sharpness of the image is subjective, the degree of sharpness SL is found in the 
same way for image data having an optimum sharpness obtained experimentally, this value is set as an ideal sharpness 
bLopt, and a degree of edge emphasis Eenhance is found from the relation 

Eenhance = ks • (SLopt-SL)* * (1/2) (41) 

in a step SA520. 

Herein, ks varies based on the magnitude of the image, and when the image data comprises height dots and width 
dots in the vertical and horizontal directions as shown in Fig. 24, ks is found from 

ks = min (height, width)/ A ( 42 ) 

Herein, min (height, width) denotes the smaller of height and width, and A is the constant "768-. It will be understood 
that these relations are obtained from experiment, and may be modified as appropriate. However, good results are 
30 basically obtained by making the degree of emphasis larger the larger the image. 

When the degree of edge emphasis Eenhance is found in this manner, it is compare* with a predetermined thresh- 
old value .n a step SA530, and it is determined whether edge emphasis is necessary. If it is deemed to be necessary 
edge emphasis processing is performed on all picture elements in a step SA540. 

In edge emphasis processing, a luminance Y' after emphasis relative to the luminance Y of each picture element 
3S before emphagis is computed by: 



Y' = Y+Eenhance • (Y-Yunsharp) 



(43) 



Herein, Yunsharp is unsharp mask processing relative to image data of each picture element. Unsharpmaskprocessinq 
will now be described. Figure 25 shows an example of a 5x5 picture element unsharp mask. In this unshaip mask 41 
the central value "100' is a weighting of a picture element Y (x.y) in matrix image data, which is used for multiplication 
with weighting corresponding to a numerical value in the grid of the mask for edge picture elements. When this unsharp 
mask 41 is utilized, multiplication is performed based on the computational equation 



Yunsharp (x.y) = (1/396) £ (MijXY(x+l, y + j) 



L.i 



(44) 



In equation (44). '396' is a total value of weighting coefficients, and it is the total value of each grid division in unsharp 
masks of different size. Mij is a woighting coefficient writton in a grid division of the unsharp mask, and Y (x y) is imago 
data for each picture element, ij is expressed in horizontal and vertical coordinate values for the unsharp mask 41 

The meaning of the edge emphasis computation based on equation (43) is as follows. As Yunsharp (x y) is added 
by making the weighting of edge picture elements lower than that of main picture elements, the result is "unsharp" 
image data. Images which are made unsharp in this way have the same meaning as those subjected to a low pass 
filler. Therefore. 'Y(x. y)-Unsharp(x.y)" means tow frequency components are removed from the total components 
which has Ihe same meaning as applying a high pass filter. If high frequency components which passed through the 
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high pass filter are multiplied by the edge emphasis Eenhance and added to "Y(x. y)", the high frequency components 
are increased in direct proportion to the edge emphasis Eenhance, and the edges are thereby emphasized Considering 
the situation when edge emphasis is needed, edge emphasis refers to the edge of the image, and it therefore may be 
computed only when there ts a large difference of image data between adjacent picture elements. If this is done there 
is no need to compute an unsharp mask for most image data which is not edge parts, so the amount of processing is 
vastly reduced. K M 

In the actual computation, if we write: 



'° A = Y-Y' (45) 

from the luminance Y' after emphasis and the luminance Y before emphasis. R' G' B' after conversion may be computed 



15 

FT- R+de1ta 
G'= G+de1ta 

20 B'- B+de1ta 
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(46). 



Therefore, in this edge emphasis processing in the steps SA51 0-SA530, the need for edge emphasis and its extent 
are determined, and image processing is performed when it is determined to be necessary in the step SA530. A process- 
ing unit comprising hardware and software is provided to perform those functions. 

It is determined whether to perform image processing regarding contrast correction, lightness compensation sat- 
uration emphasis and edge emphasis. However, it is not absolutely necessary to make a choice as to whether or not 
to perform image processing. Specifically, an emphasis degree may be set for each, and image processing performed 
with the set emphasis degree. Of course, in this case also, the contents and extent of image processing which should 
be performed are determined, and the processing is performed. 

Next, the operation of this embodiment having the aforesaid construction will be described 

A photographic image is read by the scanner 11, and printed by the printer 31. Specifically, when the operating 
system 21 a is running on the computer 21 . the image processing application 21 d is started, and reading of the photo- 
graph is started by the scanner 11 . After the read image data has been assimilated by the image processing application 
21d via the operating system 21a, the picture elements to be processed are set in initial positions. Next the edginess 
is determined based on equations (1)-(3) in the step SA110. and the edginess is compared with a threshold value in 
the step SA120. When the edginess is large, it is determined that the picture element to be processed is an edge 
picture element, and image data for the corresponding picture element is stored in the work area in the step SA130 
In the step SA140, it is determined whether or not picture elements to be processed are to sampled uniformly, and if 
so, in the step SA150, image data for these picture elements is stored in the work area. The above processing is 
repeated until it is determined to have been performed for all picture elements in the step SA1 60 while picture elements 
to be processed are displaced in the step SA150. 

When this has been performed for all picture elements, image data for picture elements determined to be those 
of the object are stored in the work area. Therefore, even if the situation of a photographic image read from the image 
data in this work area is determined, the nature of the image is not misinterpreted due to the effect of the background, 
etc. According to this embodiment, image data was stored in the work area, but from the viewpoints of memory capacity 
and processing time, it is not absolutely necessary to store image data itself in the work area. Specifically, histograms 
of luminance distribution or saturation substitute value distribution for picture elements determined to be those of the 
object are generated, and histogram information may therefore be pre-stored in the step SA140. 

When contrast correction and luminance compensation are performed automatically, a histogram of luminance 
distribution is lound in the step SA140 or the step SA310, a parameter for expansion processing is determined based 
on equations (8), (9) in the step SA320, and a parameter for luminance compensation is determined based on equations 
(10)-(1 3) in the step SA330. Those parameters are compared with a predetermined threshold value in the step SA340, 
and if it is determined that image processing should be performed, the luminance is converted based on the aforesaid 
parameter in the step S A350. In this case, the luminance conversion table shown in Fig. 1 6 may be generated to reduce 
the computing amount, and image data may be converted based on equations (14)-(16). 

Subsequently, the processed image data may be displayed on the display 32 via the display driver 21c and if it is 
satisfactory, it is printed by the printer 31 via the printer driver 21b. In other words the printer driver 21b inputs RGB 
gradation data with emphasized edges, performs rasterization corresponding to the print head area of the printer 31 
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after a predetermined mapping, color converts rasterized data from RGB to CMYK, converts CMYK gradation data to 
binary data, and outputs it to the printer 31 . 

Due to the above processing, photographic image data read via the scanner 11 is automatically subjected to the 
optimum contrast correction and lightness compensation, displayed by the display 32, and printed by the printer 31 
In other words, it is determined whether or not contrast correction and lightness compensation are necessary based 
on the object part of the corresponding photographic image, and if it is necessary, the optimum degree of image process- 
ing is performed. When object picture elements are sampled, although edginess is required to determine parameters 
it is not necessary to find the edginess for all picture elements. The edginess may be determined for sampled picture 
elements, and a determination made as to whether or not the picture elements are those of the object. 

The invention is not limited to this contrast correction and luminance compensation. In t he case also of saturation 
emphasis and edge emphasis, picture elements for which there is a large variation amount are determined to be those 
of the object. The content and extent of image processing are determined based on image data for object picture 
elements, and the necessary processing is performed. 

Hence, the computer 21 which is the core of image processing calculates the edginess, which is the image variation 
amount, from data for adjacent picture elements in the step SA110, selects only images which have a large edginess 
and determines them to be object picture elements in the steps SA120, SA130, and calculates optimum parameters 
for performing contrast correction and lightness compensation from image data for object picture elements in the steps 
SA310-SA330. An image processing indicator can therefore be determined based on image data for object picture 
elements, and optimum image processing can be performed automatically. 

Embodiment 2 
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Next, an embodiment of an image processing apparatus will be described wherein an image processing indicator 
specifying unit comprises a feature amount uniform sampling unit, and a predetermined weighting is applied after 
sampling without a large computational amount in the sampling stage so as to automatically perform optimum imaqo 
processing. 

Fig. 26 is a block diagram of an image processing system to which the image processing apparatus according to 
one embodiment of this invention is applied. The actual hardware construction may be similar to the system shown in 

Fig. 2. 

In Fig 26, an image reader 10 outputs photographic image data represented as dot matrix picture elements to an 
image processing apparatus 20B, and the image processing apparatus 20B performs image processing after deter- 
mining a degree of emphasis via a predetermined process. The image processing apparatus 20B outputs the processed 
image data to an image output device 30, and the image output device 30 outputs the processed image in dot matrix 
picture elements. Herein, the image data which the image processing apparatus 20B outputs is obtained by uniformly 
sampling fealyre amounts from picture elements by a predetermined criterion, reevaluating them with a predetermined 
weighting, and processing them with an emphasis degree determined according to the reevaluated feature amounts 
Therefore, the image processing apparatus 20B comprises a feature amount uniform sampling unit which uniformly 
samples feature amounts, a feature amount weighting revaluation unit which reevaluates sampled feature amounts 
with a predetermined weighting, and a processing unit which performs image processing with the degree of emphasis 
according to the reevaluated feature amounts. 

Specifically, the determination of the object and accompanying image processing are performed by an image 
processing program in the aforesaid computer 21 corresponding to a flowchart shown in Fig. 27. In the flowchart shown 
in the figure, image processing is performed to adjust the contrast of the image. After sampling luminance which is a 
feature amount while uniformly thinning out picture elements from the whole image in a step SB110, this feature amount 
is reevaluated by applying a predetermined weighting in a step SB120, and image processing to adjust luminance is 
performed in steps SB130-SB160. 

In the step SB110, the luminance of each picture element in dot matrix image data in the horizontal and vertical 
directions is found as shown in Fig. 24, and a histogram is generated, in this case, if the processing is applied to all 
picture elements it can be said to be precise, but as the summation results are reevaluated by weighting, it does not 
necessarily have to be precise. Therefore picture elements wherefrom the luminance has been sampled to within the 
limits of a certain error are thinned out, processing amount is reduced, and the process is thereby speeded up. Ac- 
cording to statistical error, an error for a sample number N can generally be represented by 
1/(N**(1/2)) where ** represents involution. 

Therefore in order to perform processing with an error of around 1%, N = 10000. 

Herein, the bitmap screen shown in Fig. 24 is number of (width)x(height) picture elements, and a sampling period 
ratio is given by 
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ratio = min(width. height)/ A+1 (47) 

min (width, height) is the smaller of width and height Herein, and A is a constant. The sampling period ratio mentioned 
5 here expresses how frequently to perform sampling in numbers of picture elements, and the mark O in Fig. 28 shows 

the case where the sampling period ratio = 2. In other words, one picture element is sampled every two picture elements 

in the vertical and horizontal directions, so sampling is performed every other picture element. The number of sampling 

picture elements in 1 line when A = 200 is as shown in Fig. 29. 

As is clear from the figure, except for the case when the sampling period ratio = 1 when sampling is not performed, 
io at least 100 picture elements are sampled 

when there is a width of 200 picture elements or more. 

Therefore, when there are 200 or more picture elements in the vertical and horizontal directions (100 picture elements) 
x (100 picture elements) = 10000 picture elements are sampled and the error is 1% or less. 

The reason for taking min (width, height) as a basis is as follows. For example, as shown by the bitmap in Fig. 30 
(a), if width» height and the sampling period ratio is determined by width which is the longer direction, only two lines 
of picture elements, i.e. the top edge and bottom edge, can be sampled in the vertical direction as shown in (b). 
However, if the sampling period ratio is determined based on the smaller of the two as min (width, height), thinning 
which includes the middle part can be performed even in the lesser, vertical direction as shown in (c). In other words, 
sampling with a predetermined number of samplings can be guaranteed. 

Here, the feature which is sampled with thinning of picture elements is luminance. As described above, according 
to this embodiment, the data handled by the computer 21 is RGB gradation data, and it does not directly have luminance 
values. To calculate luminance, a color conversion to the Luv color specification space could be performed, but this is 
not a good solution due to the problem of computing amount. 

For this purpose, the aforesaid equation (28) which calculates luminance directly from RGB and is used for television, 
25 etc., is utilizod here. 

Also, the luminance histogram is not summed for the whole image, but the input image is divided into 3 horizontal 
blocks and 5 vertical blocks, i.e. a total of 1 5 blocks, and the summation carried out for each block, as shown in Fig. 
31 . According to this embodiment there are 15 blocks, however there is of course no restriction on the block division 
used. 

30 in particular, with printer drivers, etc., image data is received from an application in block units, and these blocks may 
also be used to demarcate areas for weighting. 

The reason for summing in blocks in this way is to reduce the processing amount required. As reevaluation with 
weighting is performed in the step SB 120 it is not absolutely necessary to perform the summation for each block, and 
the summation may instead take the form of a histogram which considers the weighting for each selected picture 

35 element Moreover, as the weighting changes relative to the summation result regardless of blocks, the summation 
can also be made with one histogram using a weighting depending on the block. Fig. 32 is a figure showing an example 
of luminance distribution of the block Bi. 

When the summation is to performed for each block, a reevaluation is made by weighting according to area in the 
step SB120. Fig. 33 and Fig. 34 show examples of weighting each block. In the case of an ordinary photographic 

40 image, the object being photographed is usually in the center. In this sense, the feature amount should be evaluated 
placing more on the center part of the image-data. On the other hand, in the case of a souvenir photo taken in front of 
a building, the person being photographed is generally in bottom center Specifically, the person is usually in the lower 
part of the image with respect to height above ground. In this case therefore, the feature amount should be evaluated 
by weighting the lower middle part of the image. Fig. 33 shows an example of the former case, and Fig. 34 shows an 

4 $ example of the latter. 

If the weighting of each block = Wi (i) (i = 1-15) and the weighted, reevaluated luminance distribution is DY. 



SO i=l- 15 



ss then: 



SP = £ Wi ... (48) 
is 

Ki = VWSP (49) 
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DY = 2 Ki * dYi - • - - (50) 

i=l-15 

After the histogram of the reevaluated luminance distribution is obtained in this way, the intensity of image process- 
ing is calculated from this feature amount. In other words, the width for increasing contrast is determined This is done 
exactly as described above, i e. a maximum value ymaX. minimum value ymin and median ymed are acquired in the 
step SB1 30. the expansion factor a and offset b are found in the step SB140. and processing to generate a conversion 
table is performed in the step SB150. In the step SEH60, for image data (R0. GO. BO) for all P irt..m »ip mc „tc , he 
processing to obtain .mage data (R. G, B) after conversion is repeated while referring to conversion tables correspond- 
ing to equations (14)-(16). r 

A processing unit is provided comprising the hardware and software to perform the steps SB130-SB160 
It will moreover be understood that although according to this embodiment, the image processing described is that of 
contrast correction and lightness compensation, the invention may be applied in exactly the same way to other imaae 
emphasis processing. a 

In the above processing, the feature amount was reevaluated by weighting according to a position in the image 
However the weighting criterion is not limited to this, and various other types of criterion are possible As an example 
Fig. 35 shows a flowchart tor the case where the object being photographed is detected from the image variation 
amount, and the weighting is varied. 

A step SB210 replaces the above step SB110, and luminance is summed while thinning out picture elements 
uniformly. However, instead of summation only of luminance, edginess can also be summed as shown below 

In the input image it can be said that contrast of the background changes gradually; on the other hand, the object 
is sharp and so there is an intense change of luminance. Therefore, a difference of density is found between one picture 
element and surrounding picture elements as shown in Fig. 36. This density difference is taken as the edginoss of the 
picture element considered. This density difference may be computed by appfying a filter. Figure 37 (a)-(f) show several 
examples of such a filter, and show weighting coefficients when the luminance is weighted for a specific picture element 
and eight surrounding picture elements. Here, in the case of (a), the weighting is found for nine picture elements so 
mne multiplications and eight additions are necessary to calculate the edginess of each picture element When the 
image becomes large, it is impossible to ignore this computing amount, so five multiplications and four additions are 
performed m (b). (c). three multiplications and two additions are performed in (d). (e) and one addition are performed 
in (f). 

^ In these examples, the picture element in question is compared only with surrounding picture elements With the 
unsharp mask", the sharpness of the considered picture element may be found by using a wider range of image data 
However, as^dginess in this embodiment is only a tool to evaluate the weighting per block, even the filters of these 
examples of reduced computation amount give sufficiently good results. 

The summation of edginess may be performed by summing the edginess of picture elements for each block 
Alternatively, when the absolute value of this edginess is larger than a predetermined threshold value absolutely, the 
picture element is determined to be an edge picture element, and the total number of edge picture elements for each 
block is summed. When the edginess in each block is written as ERi (i =1-15), this total number SE is given by 



SE = £ ERi . . . (51) 

45 i=l-15 

so the weighting coefficient KEi itself may be expressed as 
so KEi = ERi/SE ( 52 ) 
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Therefore the luminance distribution DY which is reevaluated by weighting may be calculated 

DY = £ KEi * dYi ... (53) 



as: 
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Also, if the total number of edge picture elements in each block is ENi (i =1 -15), the total number SE is: 

3E = J! ENi --- (54) . 

i-1.15 

The weighting coefficient KEi itself is expressed as 

KEK--ENi/SE 



"(55)7 



so the luminance distribution DY which was reevaluated by equation (53) can be obtained. In any case, the luminance 
distribut on DY is reevaluated based on the computational equation (53) in the step SB220. 

In this example the luminance of picture elements determined to be edge picture elements is not sampled. Instead, 
the edgrncss and total number of edge picture elements is merely used for determining block weighting coefficients 
In other words instead ol summing the feature amounts for picture elements having a specific property (edginess), an 
average feature amount which is not uneven can be obtained for the block. 

If the luminance distribut bn is reevaluated in this way, the contrast may be increased and the 'ightness may be 
modified by the piocessing ol the aloresaid steps SB130-SB16U 

In view of the fact that the original object is often that of a person, a reevaluation may also be made by placing 
more weight on picture elements with skin color. Fig. 38 shows a flowchart wherein attention is paid to a specific color 
to determine block weighting coefficients. 

In a step SB310 corresponding to the step SB110, luminance is summed by the same thinning process, it is de- 
termined whether or not picture elements appear to have skin color based on the chromaticity of each picturo clement, 
and all picture elements having skin color are then summed. For chromaticity, x-y chromaticity is calculated for each 
picture element. Now, if 

30 r=R/(R+G+B) (56) 

g = G/{R+G+B) (57) 

35 when the^JGB gradation data of object picture elements in the RGB colorimetric system is (R, G, B). the following 
relationships exist between chromaticity coordinates x, y in the X, Y, Z colorimetric system: 



x = {1.1 302+1. 6387r+0.6215g)/(6. 7646-3.01 57r-0.38579) (58) 

y = (0.0601+O.9399r+4.5306g)/(6.7846-3.0157r-0.3857g) (59). 

Herein, as chromaticity represents an absolute proportion of a color stimulation value without it being affected by 
45 lightness, it may be said that the nature of an object can be determined from the chromaticity of its picture elements 
Since 



0,35<x<O.40 



(60) 



O.33<y<0.36 (6!) 

in the case of skin color, it may be considered that if a picture element is within this range when chromaticity is deter- 
mined for each picture element, that picture elements shows a persons skin, and the number of skin color picture 
elements in the block is increased by one. 

After the number of skin color picture elements is obtained in this way, in the next step SB320, weighting coefficients 
are determined as in the case of edge picture elements described above, and the luminance distribution DY is reeval- 
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uated. Specifically, if the number of skin color picture elements in each block is written as CNi (i =1-15) the total number 
SC of such elements is- 

SC = £ CNi ... (52) 

i=3— 15 

Therefore the weighting coefficient KCi is expressed by 



KC i - C fsifrSC- 



(63), 

and the luminance distribution DY reevaluated with this weighting may be calculated by 

DY = £ KCi*dYi . . . ( 64 ) . 

15 

Also in this example, the luminance of picture elements determined to be skin cobr picture elements is not sampled 
the total number of skin color picture elements merely being used for determining block weighting coefficients Therefore 
an average feature amount that is not uneven can be obtained for the block. In this case too, after the luminance 
d.stnbution has been reevaluated in this way the contrast may be corrected and lightness compensated by the process- 
ing of the aforesaid steps SB130-SB160. In the case of the photograph shown in Fig. 39, a girl appears in the center 
and p.cture dements of tho face, arms and logs arc determined to bo skin color picture elements Of course the 
cnromaticity may also be found and picture elements summed for other colors. 

Until now, the weighting coefficient was determined by one factor, but the importance of each factor may also be 
added and the above processing applied repeatedly. When the weighting coefficient of each block Bi (i =1-15) is Tji 
for a factor j (1 = position in image. 2 = edginess, 3= skin color picture element number), the weighting Tji distributed 
among blocks for each factor is a temporary weighting. 

Sj = X Tji - - (65) 

i=l-15 

and 

Kji = Tji/Sj (66) 
then the real weighting coefficient Ki in the block Bi is given by 
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Ki = X Aj*Kji . . . (67) 



Aj is a coefficient to represent the importance of each factor, and is suitably determined so thai the total number is 1 
If skin color is emphasized as an example, the settings A1 = O.2. A2 = O.2. A3 = 0.6 are possible. 

Next, the effect of this embodiment having the aforesaid construction will be described. First, a description will be 
given along the lines of the previous embodiment. 

A photographic imago is road by the scanner 11 , and printed by tho printer 31 . Specifically, when tho operating 
system 21a is runn.ng on the computer 21. the image processing application 21 d is started, and reading of the photo- 
graph ,s started by the scanner 1 1 . After the read image data has been assimilated by the image processing application 
21d, the luminance of picture elements is summed while thinning in the step SB11 0. The summed luminance distribution 
dY. is reevaluated in the step SB120 based on determined weightings corresponding to the position of each block 
shown ,n F.g. 33 and F.g. 34. and ymax, ymin and ymed are calculated in the step SB130 based on the reevaluated 
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In the next step SB1 40, the slope a and offset b which are emphasis parameters are computed based on equations 
(8) or (9), the y value of the 7 correction required for lightness compensation is calculated based on equations (10)- 
(13). and the conversion data shown in Fig. 16 is generated in the step SB150. Finally, in the step SB160. the image 
data for all picture elements is converted by referring to this conversion table. 

Using the weightings shown in Fig. 33, the weighting is higher for blocks near the center, so the summed luminance 
distribution is also weighted more heavily the closer it is to the center. Assume for example that a person is photographed 
at night using flash. Even if a good overall luminance distribution is obtained for the person as shown in Fig. 40(a) due 
to the effect of the flash, the part surrounding the person is dark, and here the luminance distribution is shifted towards 
the dark side as shown in (b). In this case if an average were merely taken, a luminance distribution shifted towards 
the dark side overall would still be obtained as shown in (c), and if the contrast were corrected and the lightness 
compensated, the image would only be made too light overall and would not be satisfactory. 

However, if more weighting is given to the center block as shown in Fig. 33, a luminance distribution DY is obtained 
which is strongly affected by the luminance distribution in the center of the image as shown in Fig. 40(d). Hence, the 
intensity of image processing based on this distribution is no longer a matter of over-emphasizing contrast and over- 
compensating lightness. 

Conversely, if a person is photographed with a backlight, the face will be dark, and the background will be light, 
so a good luminance distribution will not necessarily be obtained overall. Yet even in this case, by giving more weight 
to the luminance distribution of center blocks where the face is dark as shown in Fig. 33, the dark luminance distribution 
is reflected, and image processing is performed to increase contrast and compensate lightness. 

Due to the aforesaid processing, photographic image data read via the scanner 11 is processed automatically with 
optimum intensity, displayed on the display 32, and printed by the printer 31. 

The computer 21 which is the core of image processing sums the luminance distribution which a feature amount 
for each area while unilormly selecting picture elements in the step SB110. In the step SB120, a revaluation is per- 
formed with weightings determined for each area, and a luminance distribution strongly influenced by the intrinsic 
luminance distribution of the object can thus bo obtained while uniform sampling is performed. The intensity of imago 
processing is determined based on this luminance distribution in the steps SB130-SB150, and the image data is con- 
verted in the step SB1 60. Hence, image processing is performed with optimum intensity while the amount of processing 
is reduced. 



30 Embodiment 3 



Next, an embodiment will be described wherein the aforesaid image processing indicator specifying unit comprises 
an evaluation unit that obtains feature amounts based on a plurality of predetermined criteria, and the aforesaidp recess- 
ing unit which can convert image data by a plurality of methods, uses feature amounts according to the method. 

Fig,-41 shows a block diagram of an image processing system which applies the image processing apparatus 
according to one embodiment of this invention. A typical hardware constructbn is similar to the system shown in Fig. 2. 

In Fig. 41, the image reader 10 outputs photographic image data as dot matrix picture elements to the image 
processing apparatus 20C. When plural image processings are applied, the image processing apparatus 20C deter- 
mines plural feature amounts based on the optimum criterion for each type of processing, and carries out each type 
of processing using the most appropriate feature amount. The image processing apparatus 20C outputs the processed 
image data to the image output device 30, and the image output device outputs the processed image in dot matrix 
picture elements. 

The image processing apparatus 20C first sums the image data according to plural evaluation criteria, and thereby 
obtains plural feature amounts. In this sense, the image processing apparatus 20C comprises an evaluation unit, and 
as it performs image processing based on feature amounts which have been selected according to the image process- 
ing content, it may be said to further comprise an image processing unit. 

The acquisition of image feature amount of image and associated image processing are performed by the computer 
21 with an image processing program corresponding to the flowchart shown in Fig. 42. The flowcharl shown in the 
figure corresponds to the preceding stage in the image processing program, and processing is performed to sum image 
data according to plural evaluation criteria. 

According to this embodiment, the case will be described where two evaluation criteria are used. Common points 
are that in both cases, not all picture elements are considered, picture elements are sampled according to predeter- 
mined criteria, and luminance is summed for the sampled picture elements. 

What is different is that in one method, picture elements are sampled uniformly, whereas in the other method, edge 
picture elements are selected for sampling. Luminance summation results are described hereafter, but here it should 
be noted that plural feature amounts are obtained according to different evaluation criteria by changing the sampling 
method. 

Uniform sampling means that luminance is summed for all picture elements in the image, and the luminance distribution 
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is determined for the image as a whole. A feature amount is thereby obtained which is useful as a reference when a 
scene photograph is dark overall or contrast is narrow In the other method, as edge picture elements are a sharp part 
of the .mage, the luminance is summed for picture elements related to the object in the image. Therefore provided that 
the object is sufficiently light even if the background is dark, a feature amount is obtained which is useful as a reference 
when the .mage is sufficiently light. According to this embodiment, these feature amounts are selected automatically 
according to the image processing method. 

Referring to the flowchart of Fig. 42. in this summation processing, object picture elements are main scanned in 
the horizontal direction and auxiliary scanned in the vertical direction for image data comprising dot matrix picture 
elements, and displaced, as shown in Fig. 24, and the summation is performed by determining whether or not each 
picture element scanned is to be included in the sampling. 

Mrsi. m a step SC110. the edginess of each picture element is determined. Specifically, the determination of edq-~ 
mess may be the same as that of the aforesaid step SA110. 

In the step SC120. the edginess is compared with the same threshold value, and it is determined whether or not 
the var.at.on ,s large If as a result of comparison it is determined that edginess is large, it is determined that this picture 
element .s an edge picture element, and the image data for the picture element is sampled in a step SC 130 and stored 
in the work area The work area may be a RAM in the computer 21 . or it may be a hard disk 22 

In this emoodiment. the object is sampled based on edginess, but of course the method of sampling the object is 
not limited to th.s For example, the chromaticity of each picture element may be found, and picture elements for which 
the chromaticiiy is withm a predetermined range can be sampled as the object. 

Specifically, the processing of the step SC110 and slepSC120 is substituted by the processing of a step SC115 
and a step SCI 25 .especUvely. the chromaticity for each picture element is calculated in the step SC115, and in the 
step SC1 25. ,t rs determined whether or not the x-y chromaticity that was converted based on RGB gradation data for 
each picture element is within the range of skin color. If it is skin color, the image data of the picture element is sampled 
in the step SCI 30 and also stored similarly to work area. 

On the other hand, in parallel with the aforesaid determination of edginoss. it is determined in a step SC1 40 whether 
°. r n0 L!!^ ,C,Ure etemen ' iS ,0 b ° sam P ,ed b V unifo ™ sampling. Uniform sampling is identical to the above In the 
step SC140 it is determined whether or not the picture element is to be sampled, and if so, the picture element is 
sampled in a step SC1 SO. It will be understood that the sampling of image data in the steps SC1 30 SC 150 means the 
summation of luminance based on this image data 

To perform this processing for all picture elements of the image data, a picture element to be processed is displaced 
in a step SC160. and the processing is repeated until it is determined in a step SC170 that processing of all picture 
elements has finished. 

Thereafter, a feature amount is obtained by using the summation result Dist_ave obtained by uniform sampling 
and the summation result Dist.edg obtained by edge picture element sampling according to the intended image 
prccessingjnethod. and sooptimum image processing basedon this feature amount may be performed. As an example 
ng. 43 shows a flowchart for performing expansion of contrast and lightness compensation. The basic method of 
increasing contrast according to this embodiment is as described hereabove. The correction parameters are generated 
in steps SC31 0-SC330. Then, it is determined in a step SC340 whether contrast correction and lightness compensation 
c^o n C n CeS f " " iS de,ermined tnat are - conversion of image data is performed in a step SC350 In this step 
5>C350, reference is made to a conversion table corresponding to equations (14)-(16) for image data (RO GO BO) for 
all picture elements, and the process of obtaining image data (R.G.B) after conversion is repeated ' 

However in th.s case the luminance summation results are used as appropriate, and a feature amount is obtained 
comprising both ends and the median of the luminance distribution. Contrast correction and lightness compensation 
are applied using this feature amount, but this is not the only specific example of image processing, and there are also 
arsho^^Rg^ " n,S ***** ^ ^ eX3mpte ' emphasis image processing may be performed 

In this case, the evaluation unit comprises a first stage of a program for summing image data by the uniform 
sampling method (step SA410) up to the acquisition of a feature amount which is a saturation emphasis indicator S 
step SA420) and hardware for implementing this first stage of the program. The image processing unit comprises the 
atter stage of the program for performing conversion of image data (step SA440). and hardware for implementing this 
latter stage of the program. 

Regarding contrast correction, lightness compensation and saturation emphasis, it is determined whether to per- 
form image processing in each case. However, it is not absolutely necessary to make a choice as to whether or not to 
perform image processing. Specifically, a degree of emphasis degree is set for each, and image processing may be 
performed with the set degree of emphasis. 9 ^ 
Next, the operation of this embodiment having the aforesaid construction will be described 
A photographic image is read by the scanner 11. and printed by the printer 31. Specifically, when the operating 
system 21 a is running on the computer 21 . the image processing application 21d is started, and reading ot the photo- 



26 



EP 0 891 075 A2 



10 



IS 



20 



25 



30 



3S 



40 



graph is started by the scanner 1 1 . After the read image data has been assimilated by the image processing application 
21d via the operating system 21a, the picture elements to be processed are set in initial positions Next, the edginess 
is determined based on equations (1 )-(3) in the step SC110. and the edginess is compared with a threshold value in 
the step SC120. When the edginess is large, it is determined that the picture element to be processed is an edge 
picture element, and image data for the corresponding picture element is stored in the work area in the step SC130. 
In the step SC140, it is determined whether or not picture elements to be processed are to sampled uniformly and if 
so, in the step SC150, image data for these picture elements is stored in the work area. The above processing is 
repeated until it is determined to have been performed for all picture elements in the step SC170 while picture elements 
to be processed are displaced in the step SC160. 

According to this embodiment, image data was stored in the work area, but from the viewpoints of memory capacity 
and processing time, it is not absolutely necessary to store image data itself in the work area. Specifically, histograms " 
of luminance distribution or saturation substitute value distribution for picture elements determined to be those of the 
object are generated, and histogram information may therefore bepre-stored in the steps SC120, SC150. 

When summation has been performed on all picture elements, luminance distribution histograms are found for the 
summation result Dist_ave obtained by uniform sampling and the summation result Dist_edg obtained by edge picture 
element sampling in the step SC310. A parameter for expansion processing is determined based on equations (8), (9) 
in the step SC320, and a parameter for luminance compensation is determined based on equations (10)-(13) in the 
step SC330. These parameters are compared with a predetermined threshold value in the step SC340, and if it is 
determined that image processing should be performed, the luminance is converted based on the aforesaid parameter 
in the step SC350. In this case, the luminance conversion table shown in Fig. 16 may be generated to reduce the 
computing amount, and image data may be converted based on equations (14)-(16). 

Subsequently, the processed image data may be displayed on the display 32 via the display driver 21c, and if it is 
satisfactory, it may be printed by the printer 31 via the printer driver 21b. Specifically, the printer driver 21b inputs RGB 
gradation data with emphasized edges, performs rasterization corresponding to the print head area of the printer 31 
after a predetermined mapping, color converts rastcrized data from RGB to CMYK, converts CMYK gradation data to 
binary data, and outputs it to the printer 31 . 

Due to the above processing, photographic image data read via the scanner 11 is automatically subjected to the 
optimum contrast correction and lightness compensation, displayed by the display 32, and printed by the printer 31. 
Specifically, plural feature amounts are obtained using plural evaluation criteria, and optimum image processing is 
performed with different feature amounts according to the image processing methods of contrast correction or lightness 
compensation. 

However, the invention is not limited to this contrast correction and luminance compensation. In the case also of 
saturation emphasis, a feature amount is acquired by sampling saturation according to a suitable criterbn depending 
on the saturation emphasis processing, and image processing is performed based on this feature amount. In this way, 
optimum iooage processing is performed. 

Hence, the computer 21 which is the core of image processing calculates the luminance distribution based on the 
summation results sampled according to different evaluation criteria in the step SC310, obtains different feature 
amounts from different luminance distribution histograms in the steps SC320 and SC330. and converts image data 
based on these feature amounts in the step SC350. This permits optimum image processing. 

Embodiment 4 
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Next, an embodiment will be described concerned mainly with image evaluation which is an indicator for image 
processing. 

Fig. 44 is a block diagram of an image processing system which performs image processing by implementing an 
image evaluation method according to one embodiment of this invention. A typical hardware construction is similar to 
the system shown in Fig. 2. 

In Fig. 44, the image reader 10 outputs photographic image data as dot matrix picture elements to an image 
processing apparatus 20D. The image processing apparatus 20D calculates an evaluation result by summing the image 
data after predetermined processing, determines the content and extent of image processing based on the evaluation 
result, and then performs image processing. The image processing apparatus 20D outputs the processed image data 
to an image output device 30, and the image output device outputs the processed image as dot matrix picture elements. 

The image processing apparatus 20D sums the image data beforehand, and calculates an evaluation result for 
the corresponding image. The image data are summed individually using plural evaluation criteria, and are combined 
by varying the weighting according to predetermined conditions. Therefore, the image processing apparatus 20D is 
an image data evaluation unit. 

Image evaluation and associated image processing are performed in the computer 21 by an image processing 
program corresponding to a flowchart such as is shown in Fig. 45. The flowchart shown in the figure corresponds to 
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he first stage of .mage evaluation in the image processing program, and processing is performed to obtain a prede- 
termined evaluation result by summing the image data according to plural evaluation criteria. The basic method of 
image evaluation , is substantially identical to that described above, however in the following description a simple eval- 
uation result is obtained by using image evaluation options. 

According to this embodiment, two of the evaluation criteria used will be described. Common points are that in 
both cases, picture elements are thinned according to a predetermined criterion instead of considering the whole image 
and the luminance of the sampled picture elements is summed ' ' 

I'll 6 ?" 06 15 t ! ,at , w J he ? as in one case - P icture elements are sampled uniformly, in the other case, edge picture 
elements are selected^ The luminance summation results are described hereafter, but the image evaluation can be 
changed by changing the sampling method in this way t iniform sampling nf p irt.,«, »i ome nts means that Ihe luminanco 
is summed for all picture elements in the image, and the luminance distribution is determined for the image as a whole 
The evaluation is therefore useful as a reference when a scenic photograph is dark overall or contrast is narrow. In the 
.S'oK , 9B T UTG el£ T n,S 3 Sharp Part ° f ^ image ' ,he lumina ™* is summed for picture elements 
iTtllt , ? m f T 96 F ° r eXample ' prOVided that the obiect is light even if the background is 

Sl^nSS IT ,S ° b,a,ned where the ima 9 e * Efficiently light. According to this embodiment, the image is 
determined by suitably combining two evaluation criteria, i.e. a criterion selected by the operator or automatic process- 
is m^ i ™ 9n H W Tl fl0WChart0, Fi9> 45, S,SPS SD110 - SD170 - ima 9 eda ^comprisingdot matrix pictureelements 
is main scanned in the horizontal direction and auxiliary scanned in the vertical direction as shown in Fig 24 as de- 
scribed in the aforesaid embodiment, and image data for edge picture elements and uniformly sampled picture elements 
is stored in the work area. 

After luminance is summed for all picture elements concerned by these various sampling methods, image evalu- 

TnZ JTr? 3 T 1 8 S,6P S P 1B ° Fi9UrC 46 ShOWS ™ ima9S eva,uati °" °P«°n input screen displayed on the 
display 32. Three choices are available: portrait, scenic photograph and automatic setting 

i< nh£j£ST F u- 47 ' W ° i9htin9s must bo ad ' us,od in ordor to 9 c "°'ato a histogram to evaluate the image which 
» obtained by combining a luminance histogram obtained by uniform sampling and a luminance histogram obtained 

Dy eage sampling. 

When the weighting coefficient k is adopted, a summation result Dist _ Sum for evaluation is obtained from the 
uniform sampling summation result Dist.ave and the edge sampling summation result Dist.edg by the relation: 



Dist _ Sum = k x Dist _ edg+ (1-k) xDist _ ave 



(68V 
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SnrlTr^. W ^ i9htin9 C ° effi f ient k approaches "0" the more the whole image is emphasized, and the closer it 
approaches^! . the more the object in the photograph is emphasized. 

As a result, after setting the options on the image evaluation option input screen shown in Fig. 46. the routine branches 
depending on the option in a step SD190. When a portrait is selected, k is set to 0.8 in a step SD192 and when a 
scenic photograph is selected, k is set to 0.2 in a step SD1 94. 

The remaining option is the 'AUTO" setting. 
In this auto setting, based on the edge picture elements sampled as described above, the image may be considered 
as a scenic photograph and the weighting coefficient approaches "0' when there are lew edge picture elements while 
P^ctuTeTemeL C ° nSidered 3S 3 and ,he wsi 9 htin 9 coefficient approaches "1* when there are many" edge 

By using the sampling number x_edg of edge picture elements and the uniform sampling number x ave. the weighting 
coefficient may be computed from: ~ ^ 8 
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k - x_edg/(x_edg + x_ave) 



(69) 



in a step SD196, and the summation result Dist.Sum used for evaluation can be obtained 

After the summation result Dist_Sum used for evaluation is obtained, image evaluation can be performed Of 

course a further determination may bo made using this summation result, and it may basically be varied as necessary 

depending on the image processing which uses the summation result. 

Subsequently, the optimum image processing is determined based on the summation results and performed As 

an example. Fig. 48 shows a flowchart to perform expansion of contrast and luminance compensation In a basic 

technique to increase contrast according to this embodiment, the luminance distribution is found based on image data 

nL!?J 3 0r . eSa emb J odimen, < and if ,his "finance distribution uses only part of the actual gradation width (255 
gradations), it is expanded. x 
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Therefore, in a step SD31 0, a histogram of luminance distribution is generated as the summation result Dist _ Sum 
from the weighting coefficient k, and the expansion width is determined in a step SD320. The expansion width deter- 
mining method is as described above. The generation of a predetermined conversion chart corresponds to the expan- 
sion width determination processing of the step S320. so image data can be modified by referring to this chart. By 
5 expanding the luminance range, not only is the contrast emphasized, but it is also very useful to adjust the luminance 
range at the same time. Therefore the luminance of the image is determined in a step SD330, and a parameter is 
generated for the correction. 

In this case, the summation results used for determining the image are used as evaluation criteria Contrast cor- 
rection and luminance compensation are performed, but specific examples of image processing are not limited to this, 
*o and therefore various summation results may be used as evaluation criteria. 

Examples of image processing are saturation emphasis or edge emphasis. In these cases, image vividness and 
image sharpness need to be evaluated. The summation may be performed by the above methods. 

It may be determined whether image processing is performed regarding contrast correction, luminance compen- 
sation, saturation emphasis and edge emphasis, but it is not absolutely necessary to make a choice as to whether or 
not to perform image processing. Specifically, a degree of emphasis is set for each, and image processing is performed 
using this set degree of emphasis. 

Next, the action of this embodiment having the aforesaid construction will be described. 

Assume that a photographic image is read with the scanner 11 , and printed by the printer 31 . When the operating 
system 21 a is running, the image processing application 21 d is started by the computer 21 , and reading of a photograph 
is started relative to scanner 11. After the read image dala has been assimilated via the operating system 21a by an 
image processing application 21 d. the picture elements to be processed are set in initial positions. Next, the edginess 
is determined based on equations (1 )-(3) in the step SD110, and the edginess is compared with a threshold value in 
the step SD120. When the edginess is large, it is determined that the picture element to be processed is an edge 
picture element, and image data for the corresponding picture element is stored in the work area in the step SD130. 
In tho stop SD140, it is determined whether or not picture elements to be processed arc to sampled uniformly, and if 
so. in the step SD150, image data for these picture elements is stored in the work area. The above processing is 
repeated until it is determined to have been performed for all picture elements in the step SD1 70 while picture elements 
to be processed are displaced in the step SD160. 

When this has been performed for all picture elements, image data sampled according to different evaluation 
criteria is stored in different work areas, and in a step SD1 30, image evaluation options are input. The operator, looking 
at the image, may select either a portrait or scenic photograph is that can be determined. If it cannot be determined 
or when it is desired to fully automate operations, the operator selects the auto setting. When a portrait is selected, 
the weighting coefficient k becomes "0.8", which puts more weight on the summation results for the edge picture 
elements. When a scenic photograph is selected, the weighting coefficient k becomes "0.2'. which puts more weight 
on unifomjly sampled summation results. When the auto setting is selected, the weighting coefficient k is set according 
to the proportion of edge picture elements. However, in any of these cases plural evaluation criteria are adopted using 
the weighting coefficient k. This permits a flexible evaluation not limited to just one criterion. 

According to this embodiment, image data was stored in the work area, but from the viewpoints of memory capacity 
and processing time, it is not absolutely necessary to store image data itself in the work area. Specifically, histograms 
of luminance distribution or saturation substitute value distribution for sampled picture elements are generated, and 
histogram information may therefore be prestored in the steps SD120, SD150. 

When contrast correction and luminance compensation are performed automatically, a weighting coefficient is 
used, a histogram of luminance distribution is found in the steps SD120, SD150, SD310, a parameter for expansion 
processing is determined based on equations (8), (9) in the step SD320, and a parameter for luminance compensation 
is determined based on equations (10)-(1 3) in the step SD330. These parameters are compared with a predetermined 
threshold value in a step SD340, and if it is determined that image processing should be performed, the luminance is 
converted based on the aforesaid parameter in a step SD350. In this case, a luminance conversion table shown in 
figure 16 may be generated to reduce the computing amount, and image data may be converted based on equations 
(14)-(16). 

Subsequently, the processed image data may be displayed on the display 32 via a display driver 21c, and if it is 
satisfactory, it is printed by the printer 31 via a printer driver 21b. In other words the printer driver 21b inputs RGB 
gradation data with emphasized edges, performs rasterization corresponding to the print head area of the printer 31 
after a predetermined mapping, saturation converts rasterized data from RGB to CMYK, converts CMYK gradation 
data to binary data, and outputs it to the printer 31 . 

The photographic image data read via the scanner 11 and printed by the printer 31 or displayed by the display 32, 
is therefore automatically subjected to optimum contrast correction and luminance compensation. More specifically, 
the image can be determined more flexibly by adopting plural evaluation criteria, and optimum image processing re- 
alized by contrast correction and luminance compensation. 
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However, the invention is not limited to this contrast correction and luminance compensation. Saturation and edq- 
iness are also sampled by plural evaluation criteria and summed, and weighting coefficients are adjusted and added 
Therefore, .mage processing is performed after a flexible determination which is not limited to a single evaluation 

crnsnon. 

,r^H^rL {he COmp " ter 21 **** is the core of ima 9 e Processing first samples image data for picture elements 

Cnteria 10 StePS SD120, SD14 °' 3nd determines *e weighting coefficient k in the 
trZL k f u ! * ' ma9e evaluation °P tions in P ut in the step SD180. It then generates a luminance dis- 

2£.nm i 7 9 S u Ummation results in the ste P SD310 usi "9 these weighting coefficients, and performs 

optimum image processing in the steps SD31 0-SD350. 
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An image processing apparatus for inputting photographic image data comprising dot matrix picture elements and 
performing predetermined image processing thereon, characterized by comprising: 

an image data acquiring unit which acquires said photographic data, 

an image processing indicator specifying unit which performs predetermined summation processing on all 
picture elements based on said acquired image data and specifis an image processing indicator, 
and a processing unit which determines image processing contents based on said specified indicator and 
performs image processing, 

wherein said image processing indicator specifying unit comprises an object determining unit which determines 
picture elements having a large image variation amount to be those of the object, and said processing unit 
determines image processing contents based on image data for picture elements determined to be those of 
the object and performs imago processing based on said determined contents. 

2. An image processing apparatus as defined in Claim 1 . wherein in said object determining unit, said image variation 
amount is determined based on a difference of image data between adjacent picture elements. 

3. An image processing apparatus as defined in Claim 1, wherein in said object determining unit, a criterion for 
determining whether or not said image variation amount is large is made to vary according to a position of said 
image. 

4. An image processing apparatus as defined in Claim 1 . wherein in said object determining unit, said criterion is set 
lower focthe center part than for the periphery of said image. 

5. An image processing apparatus as defined in Claim 3. wherein in said object determining unit, said criterion is 
determined based on a distribution of said image variation in each part of the image. 

6. An image processing method for inputting photographic image data comprising dot matrix picture elements and 
performing predetermined image processing thereon, characterized by comprising: 

an image data acquiring step for acquiring said photographic data, 

an image processing indicator specifying step for performing predetermined summation processing on all 
picture elements based on sad acquired image data and specifying an image processing indicator 
and a processing step for determining image processing contents based on said specified indicator and per- 
forming image processing, 

wherein said image processing indicator specifying step comprises an object determining step in which picture 
elements having a large image variation amount are determined to be those of the object, and in the aforesaid 
processing step, image processing contents are determined based on image data for picture elements deter- 
mined to be those of an object and image processing is performed based on the determined contents. 

An imago processing apparatus for inputting photographic image data comprising dot matrix picture elements and 
performing predetermined image processing thereon, said method being characterized in comprising: 

an image data acquiring unit which acquires said photographic data, 

an image processing indicator specifying unit which performs predetermined summation processing on all 
picture elements based on said acquired image data and specifies an image processing indicator. 
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and a processing unit which determines image processing contents based on said specified indicator and 
performs image processing, 

wherein said image processing indicator specifying unit comprises a feature amount uniform sampling unit 
which determines an image processing intensity by uniformly sampling a feature amount over a whole screen, 
and a feature amount weighting reevaluation unit which reevaluates said feature amount sampled in said 
feature amount sampling unit with a predetermined weighting, 

and in said processing unit, said image processing intensity is determined based on said reevaluated feature 
amount, and image processing is performed with said determined intensity 

I. An image processing apparatus as defined in Claim 7. wherein in said feature amount uniform sampling unit, sai d 
feature amount is sampled for picture elements selected by uniformly thinning all picture elements according to a 
predetermined criterion. 



An image processing apparatus as defined in Claim 7 or 8, wherein in said feature amount uniform sampling unit, 
a feature amount is sampled in area units obtained by dividing said image according to a predetermined criterion, 
and in said feature amount weighting reevaluation unit, a weighting is set for each of sard areas so as to reevaluate 
said feature amount. 



10. An image processing apparatus as defined in Qlaim 7-9, wherein in said feature amount weighting reevaluation 
unit, said weighting is varied according to a positional relationship determined by the position of each picture 
element in the image. 



. An image processing apparatus as defined in Claim 7-10. wherein in said feature amount weighting reevaluation 
unit, an image variation amount is calculated, and said weighting is increased in a part where said image variation 
amount is largo. 

I An image processing apparatus as defined in Claim 7-11, wherein in said feature amount weighting reevaluation 
unit, a chromaticity of each picture element is calculated, a number of picture elements having a chromaticity within 
a chromaticity range of a target for which it is desired to sample a feature amount is calculated, and said weighting 
is increased in a part where said number of picture elements is large. 

L An image processing apparatus as defined in Claim 7-12. wherein in said feature amount weighting reevaluation 
unit, temporary weighting coefficients are calculated separately based on plural factors, and said weighting coef- 
ficients are summed according to their degree of importance so as to give a final weighting coefficient which is 
then appjied. 



14. An image processing apparatus for inputting photographic image data comprising dot matrix picture elements and 
performing predetermined image processing thereon, said apparatus being characterized in comprising: 

40 an image data acquiring unit which acquires said photographic data, 

an image processing indicator specifying unit which performs predetermined summation processing on all 
picture elements based on said acquired image data and specifies an image processing indicator, 
and a processing unit which determines image processing contents based on said specified indicator and 
performing image processing, 

wherein said image processing indicator specifying unit comprises an evaluation unit which obtains a feature 
amount by inputting said photographic data, sums the image data for all picture elements and obtains feature 
amounts according to plural predetermined evaluation criteria, and in said processing unit, the image data can 
be converted by plural techniques and the feature amounts obtained by said evaluation unit are used according 
to each of said techniques. 
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15. An image processing apparatus as defined in Claim 14, wherein said evaluation unit comprises a standard for 
sampling an object in a photographic image and summing image data for object picture elements so as to obtain 
a feature amount, and in said processing unit a feature amount obtained from object picture elements is used 
when a feature amount for a center part of said image data is used in one image processing technique. 

16. An image processing method for inputting photographic image data comprising dot matrix picture elements and 
performing predetermined image processing thereon, said method being characterized in comprising: 
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an image data acquiring step for acquiring said photographic data, 

an image processing indicator specifying step for performing predetermined summation processing on all 
picture elements based on said acquired image data and specifying an image processing indicator, 
and a processing step for determining image processing contents based on said specified indicator and per- 
forming image processing, 

wherein said image processing indicator specifying step comprises an evaluation step for inputting said pho- 
tographs data, summing the image data for all picture elements, and obtaining feature amounts (plural?) 
according to plural predetermined evaluation criteria, and in said processing step, the image data can be 
converted by plural techniques and the feature amounts obtained in said evaluation step are used accordina 
to each of said techniques. 
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17. An image evaluation apparatus, characterized by comprising: 

an image data input unit for inputting photographic image data comprising dot matrix picture elements 
and an image data evaluation unit which sums image data for all picture elements according to a predetermined 
criterion, compnses plural evaluation criteria which are applied to said summation results to evaluate the image 
based on said results, and combines said evaluation results according to a predetermined weighting based 
on each of said evaluation criteria. 

18. An image evaluation method characterized in that: 

photographic image data comprising dot matrix picture elements is input, 

image data for all picture elements is summed according to a predetermined criterion, and 

the image is evaluated based on the summation results, 

said method comprising plural criteria for evaluating said summation results, and said results are combined 
according to a predetermined weighting based on each of said evaluation criteria. 
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(57) In image processing according to the prior art, 
the important part of photographic image data (referred 
to herein as the object) could not be determined and 
therefore required human participation. 

A computer 21 which is the core of image process- 
ing calculates an edginess which is an image variation 
from a differential value of data for adjacent picture el- 
ements in a step SA110, and determines object picture 
elements by selecting only images with a large variation 
in steps SA120, SA130. As optimum parameters for 
contrast correction and lightness compensation are cal- 
culated from image data for object picture elements in 
steps SA310-SA330, image processing indicators 
based on object picture elements are determined, and 
optimum image processing can be performed automat- 
ically. After summing a luminance distribution for each 
area of the image, which is a feature amount, while uni- 
formly selecting picture elements in a step SB110, a 
revaluation is performed by a weighting determined for 
each area in a step SB 120, and a luminance distribution 
strongly influenced by the luminance distribution of the 
photographed object is thus obtained with uniform sam- 
pling. After determining the intensity of this luminance 
distribution in steps SB130-SB150, the image data is 
converted in a step SB160, and image processing can 
therefore be performed with optimum intensity while re- 
ducing the processing amount 
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